Merge branch '3.5-dev' into 3.6-dev

.github/workflows/build-test.yml

@@ -92,7 +92,7 @@ jobs:
       - name: Build with Maven Windows
         if: runner.os == 'Windows'
         run: |
-          mvn clean install -pl -:gremlin-javascript,-:gremlin-python,-gremlin-dotnet,-:gremlin-dotnet-source,-:gremlin-dotnet-tests,-:gremlint -q -DskipTests -Dci
+          mvn clean install -pl -:gremlin-javascript,-:gremlin-python,-gremlin-dotnet,-:gremlin-dotnet-source,-:gremlin-dotnet-tests,-:gremlint -q -DskipTests -DskipImageBuild -Dci
       - name: Build with Maven Ubuntu
         if: runner.os == 'Linux'
         run: |
@@ -116,7 +116,7 @@ jobs:
       - name: Build with Maven Windows
         if: runner.os == 'Windows'
         run: |
-          mvn clean install -pl -:gremlin-javascript,-:gremlin-python,-gremlin-dotnet,-:gremlin-dotnet-source,-:gremlin-dotnet-tests,-:gremlint -q -DskipTests -Dci
+          mvn clean install -pl -:gremlin-javascript,-:gremlin-python,-gremlin-dotnet,-:gremlin-dotnet-source,-:gremlin-dotnet-tests,-:gremlint -q -DskipTests -DskipImageBuild -Dci
       - name: Build with Maven Ubuntu
         if: runner.os == 'Linux'
         run: |
@@ -240,10 +240,7 @@ jobs:
       - name: Build with Maven
         working-directory: .
         run: |
-          ls -la ./gremlin-go
-          touch ./gremlin-go/.glv
-          chmod +x ./gremlin-go/run.sh
-          ls -la ./gremlin-go
+          touch gremlin-go/.glv
           mvn verify -pl :gremlin-go
       - name: Upload to Codecov
         uses: codecov/codecov-action@v3

CHANGELOG.asciidoc

@@ -234,7 +234,8 @@ image::https://raw.githubusercontent.com/apache/tinkerpop/master/docs/static/ima
 [[release-3-5-5]]
 === TinkerPop 3.5.5 (Release Date: NOT OFFICIALLY RELEASED YET)
 
-
+* Changed label generation in `PathProcessorStrategy` to be more deterministic.
+* Bumped to Apache `commons-configuration` 2.8.0 to fix security vulnerability.
 
 [[release-3-5-4]]
 === TinkerPop 3.5.4 (Release Date: July 18, 2022)

README.asciidoc

@@ -34,6 +34,8 @@ TinkerPop uses link:https://maven.apache.org/[Maven] and requires `Java 11` for
 [source,bash]
 mvn clean install
 
+Please see the xref:docs/src/dev/developer/development-environment.asciidoc#building-on-windows[Building on Windows] section for Windows specific build instructions.
+
 The zip distributions can be found in the following directories:
 
 . `gremlin-server/target`

docker/scripts/build.sh

@@ -67,7 +67,9 @@ if [ -d "${TINKERMEM_PATH}" ]; then
   cd ${TINKERMEM_PATH}
 fi
 
-touch {gremlin-dotnet,gremlin-go,gremlin-dotnet/src,gremlin-dotnet/test,gremlin-python,gremlin-javascript}/.glv
+# omitting gremlin-go for now to avoid docker in docker as it builds via docker compose
+# progress to restructure and add gremlin-go back will be tracked here https://issues.apache.org/jira/browse/TINKERPOP-2737
+touch {gremlin-dotnet,gremlin-dotnet/src,gremlin-dotnet/test,gremlin-python,gremlin-javascript}/.glv
 
 # use a custom maven settings.xml
 if [ -r "settings.xml" ]; then

docs/src/dev/developer/development-environment.asciidoc

@@ -50,10 +50,8 @@ TIP: Consider using link:https://sdkman.io/[SDKMAN!] to manage Java and Maven ve
 NOTE: For those using Windows, efforts have been made to keep the build OS independent, but, in practice, it is likely
 that TinkerPop's build system will only allow for a minimum build at best. +
  +
- For Hadoop Gremlin to work on Windows, the `winutils.exe` binary needs to be downloaded and placed in local folder in
- the following file structure: `hadoop/bin/winutils.exe` and `HADOOP_HOME` must point to the `hadoop` folder as an
- environment variable. The binary for Hadoop-2.7.7 can be found https://github.com/cdarlint/winutils/blob/master/hadoop-2.7.7/bin/winutils.exe[here]
-
+ Refer to <<building-on-windows>> section for more details.
+ 
 [[groovy-environment]]
 === Groovy Environment
 
@@ -461,13 +459,33 @@ mvn -Dmaven.javadoc.skip=true --projects tinkergraph-gremlin test
 ** Generate web site locally: `bin/generate-home.sh`
 ** Publish web site: `bin/publish-home.sh <username>`
 
+[[building-on-windows]]
+== Building On Windows
+
+The following steps must be taken in order to build TinkerPop on Windows:
+
+. Install winutils for Hadoop
+* Download the latest version of link:https://github.com/cdarlint/winutils[winutils] binaries for Hadoop. The binaries `winutils.exe` and `hadoop.dll` are required.
+* Place contents of the bin folder on your local driver in the following folder structure:
+** e.g. `hadoop-3.2.2/bin/winutils.exe`
+* Set `HADOOP_HOME` to point to the `hadoop-3.2.2` folder
+* Add `%HADOOP_HOME%\bin` to your `PATH`
+. Run `mvn clean install` from root of tinkerpop
+. Follow IDE specific steps if applicable:
+* <<intellij>>
+
+You should now be able to work with TinkerPop on Windows.
+
 [[docker-integration]]
 == Docker Integration
 
 TinkerPop provides a shell script, that can start several build tasks within a Docker container. The
 required Docker images will be built automatically if they don't exist yet. Thus the first invocation
 of the Docker script is expected to take some time.
 
+NOTE: Currently, `gremlin-go` integration is not supported within this Docker container, but can be run separately with Docker.
+Please see the `Testing With Docker` section under the xref:../../../../gremlin-go/driver/README.md[Gremlin Go Driver] for details.
+
 The script can be found under `PROJECT_HOME/docker/build.sh`. The following tasks are currently
 supported:
 

docs/src/reference/gremlin-variants.asciidoc

@@ -1841,11 +1841,6 @@ To make the code more readable and close to the Gremlin query language), you can
 type implementation. To input a set into Gremlin-Go, a custom struct which implements the `gremlingo.Set` interface
 will be serialized as a set. `gremlingo.NewSimpleSet` is a basic implementation of a set that is provided by Gremlin-Go
 that can be used to fulfill the `gremlingo.Set` interface if desired.
-* The `path` data type serialization is currently incomplete as labels are represented as list of lists instead of list
-of sets. Fully functional Path serialization will be implemented when set is implemented in the next milestone. Path
-can be successfully deserialized.
-* Traversal step functions currently take `string` arguments with double quotes only. Operations using Gremlin keywords,
-such as `By(label)`, will be supported in the next milestone.
 
 [[gremlin-go-examples]]
 === Application Examples

gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/strategy/optimization/PathProcessorStrategy.java

@@ -118,7 +118,7 @@ public void apply(final Traversal.Admin<?, ?> traversal) {
                 int index = TraversalHelper.stepIndex(whereTraversalStep, traversal);
                 final SelectOneStep<?, ?> selectOneStep = new SelectOneStep<>(traversal, Pop.last, whereStartStep.getScopeKeys().iterator().next());
                 traversal.addStep(index, selectOneStep);
-                final String generatedLabel = PathProcessorStrategy.generateLabel();
+                final String generatedLabel = generateLabel(whereStartStep);
                 if (selectOneStep.getPreviousStep() instanceof EmptyStep) {
                     TraversalHelper.insertBeforeStep(new IdentityStep<>(traversal), selectOneStep, traversal);
                     index++;
@@ -191,8 +191,19 @@ public static PathProcessorStrategy instance() {
         return INSTANCE;
     }
 
-    private static String generateLabel() {
-        return IS_TESTING ? "xyz" : UUID.randomUUID().toString();
+    private static String generateLabel(final Step<?, ?> step) {
+        // use a predictable label here rather than a UUID. the UUID step label might get generated in
+        // different threads in OLAP and misbehave for something like:
+        // g.withComputer().V().as("a").out("created").where(__.as("a").values("name").is("josh")).in("created").values("name")
+        // because labels won't propagate for the SelectOneStep with processTraverserPathLabels if the labels
+        // end up being different. this bug has only recently shown itself an only under conditions i can't
+        // quite identify. it may have been masked for years by the "is.testing" flag which was pinning the
+        // label to "xyz" and for maven that flag is enabled. not sure if using the step id is "right" or "best"
+        // but it seems to give a predictable label/unique and actually one nicer label than the UUID. in the
+        // profile() the UUID almost looks like a bug if you don't understand why it's there. at least with
+        // this labelling the name of the strategy gives a hint
+        return IS_TESTING ? "xyz" : "[" + PathProcessorStrategy.class.getSimpleName() + "-" +
+                step.getId().replace(".", "+").replace("()", "") + "]";
     }
 
     private static int labelCount(final String label, final Traversal.Admin<?, ?> traversal) {

gremlin-go/Dockerfile

@@ -29,7 +29,9 @@ RUN chmod 755 /opt/docker-entrypoint.sh
 # Installing dos2unix to avoid errors in running the entrypoint script on Windows machines where their
 # carriage return is \r\n instead of the \n needed for linux/unix containers
 RUN apk update && apk add dos2unix
-RUN dos2unix /opt/docker-entrypoint.sh && apk del dos2unix
+RUN dos2unix /opt/docker-entrypoint.sh
+RUN dos2unix /opt/gremlin-server/bin/gremlin-server.conf
+RUN dos2unix /opt/gremlin-server/bin/gremlin-server.sh && apk del dos2unix
 
 ARG GREMLIN_SERVER
 ENV GREMLIN_SERVER_VER=$GREMLIN_SERVER

gremlin-go/docker-compose.yml

@@ -49,8 +49,10 @@ services:
       - RUN_BASIC_AUTH_INTEGRATION_TESTS=true
       - TEST_TRANSACTIONS=true
     working_dir: /go_app
-    command: ["./wait-for-server.sh", gremlin-test-server, "45940", "180", "bash", "-c",
-      "go install github.com/haveyoudebuggedit/gotestfmt/v2/cmd/gotestfmt@latest
-      && go test -v -json ./... -race -covermode=atomic -coverprofile=\"coverage.out\" -coverpkg=./... | gotestfmt"]
+    command: >
+      bash -c "apt-get update && apt-get install dos2unix && dos2unix ./wait-for-server.sh
+      && ./wait-for-server.sh gremlin-test-server 45940 300
+      && go install github.com/haveyoudebuggedit/gotestfmt/v2/cmd/gotestfmt@latest
+      && go test -v -json ./... -race -covermode=atomic -coverprofile=\"coverage.out\" -coverpkg=./... | gotestfmt"
     depends_on:
       - gremlin-test-server

gremlin-go/driver/README.md

@@ -119,24 +119,24 @@ Please review the [staticcheck documentation][scheck docs] for more details on i
 
 ## Testing with Docker
 
-Docker allows you to test the driver without installing any dependencies. Please make sure Docker is installed and running on your system.
-You will need to install both [Docker Engine][dengine] and [Docker Compose][dcompose], which are included in [Docker Desktop][ddesktop].
+Docker allows you to test the driver without installing any dependencies. Gremlin Go uses Docker for all of its testing. Please make sure Docker is installed and running on your system. You will need to install both [Docker Engine][dengine] and [Docker Compose][dcompose], which are included in [Docker Desktop][ddesktop].
 
 The docker compose environment variable `GREMLIN_SERVER` specifies the Gremlin server docker image to use, i.e. an image with the tag
 `tinkerpop/gremlin-server:$GREMLIN_SERVER`, and is a required environment variable. This also requires the specified docker image to exist,
 either locally or in [Docker Hub][dhub].
 
-If your OS Platform cannot build a local SNAPSHOT Gremlin server through `maven`, it is recommended to use the latest released server version
-from [Docker Hub][dhub] (do not use `GREMLIN_SERVER=latest`, use actual version number, e.g. `GREMLIN_SERVER=3.5.x` or `GREMLIN_SERVER=3.6.x`).
+Running `mvn clean install -pl gremlin-server -DskipTests -DskipIntegrationTests=true -Dci -am` in the main `tinkerpop` directory will automatically build a local SNAPSHOT Gremlin server image. If your OS Platform cannot build a local SNAPSHOT Gremlin server through `maven`, it is recommended to use the latest released server version from [Docker Hub][dhub] (do not use `GREMLIN_SERVER=latest`, use actual version number, e.g. `GREMLIN_SERVER=3.5.x` or `GREMLIN_SERVER=3.6.x`).
 
-There are 4 ways to launch the test suite and set the `GREMLIN_SERVER` environment variable depending on your Platform:
-- Execute tests via the `run.sh` script, which sets `GREMLIN_SERVER` by default. Run `./run.sh -h` for usage information (Unix/Linux - recommended).
-- Add `GREMLIN_SERVER=<server-image-version>` to an `.env` file inside `gremlin-go` and run `docker-compose up --exit-code-from gremlin-go-integration-tests` (Platform-agnostic).
+The docker compose environment variable `HOME` specifies the user home directory for mounting volumes during test image set up. This varibale is set by default in Unix/Linux, but will need to be set for Windows, for example, run `$env:HOME=$env:USERPROFILE` in PowerShell.
+
+There are different ways to launch the test suite and set the `GREMLIN_SERVER` environment variable depending on your Platform:
+- Run Maven commands, e.g. `mvn clean install` inside of `tinkerpop/gremlin-go`, or `mvn clean install -pl gremlin-go` inside of `tinkerpop` (platform-agnostic - recommended)
+- Run the `run.sh` script, which sets `GREMLIN_SERVER` by default. Run `./run.sh -h` for usage information (Unix/Linux - recommended).
+- Add `GREMLIN_SERVER=<server-image-version>` and `HOME=<user-home-directory>` to an `.env` file inside `gremlin-go` and run `docker-compose up --exit-code-from gremlin-go-integration-tests` (Platform-agnostic).
 - Run `GREMLIN_SERVER=<server-image-version> docker-compose up --exit-code-from gremlin-go-integration-tests` in Unix/Linux.
-- Run `$env:GREMLIN_SERVER="<server-image-version>";docker-compose up --exit-code-from gremlin-go-integration-tests` in Windows PowerShell.
+- Run `$env:GREMLIN_SERVER="<server-image-version>";$env:HOME=$env:USERPROFILE;docker-compose up --exit-code-from gremlin-go-integration-tests` in Windows PowerShell.
 
-You should see exit code 0 upon successful completion of the test suites. Run `docker-compose down` to remove the service containers (not needed
-if you executed `run.sh`), or `docker-compose down --rmi all` to remove the service containers while deleting all used images.
+You should see exit code 0 upon successful completion of the test suites. Run `docker-compose down` to remove the service containers (not needed if you executed Maven commands or `run.sh`), or `docker-compose down --rmi all` to remove the service containers while deleting all used images.
 
 [go]: https://go.dev/dl/
 [gomods]: https://go.dev/blog/using-go-modules

gremlin-go/go.mod

@@ -24,7 +24,7 @@ require (
 	github.com/google/uuid v1.3.0
 	github.com/gorilla/websocket v1.5.0
 	github.com/nicksnyder/go-i18n/v2 v2.2.0
-	github.com/stretchr/testify v1.7.0
+	github.com/stretchr/testify v1.7.2
 	golang.org/x/text v0.3.7
 )
 
@@ -39,5 +39,5 @@ require (
 	github.com/pmezard/go-difflib v1.0.0 // indirect
 	github.com/spf13/pflag v1.0.5 // indirect
 	github.com/stretchr/objx v0.1.1 // indirect
-	gopkg.in/yaml.v3 v3.0.0-20200615113413-eeeca48fe776 // indirect
+	gopkg.in/yaml.v3 v3.0.1 // indirect
 )

gremlin-go/go.sum

@@ -191,8 +191,9 @@ github.com/stretchr/objx v0.1.1 h1:2vfRuCMp5sSVIDSqO8oNnWJq7mPa6KVP3iPIwFBuy8A=
 github.com/stretchr/objx v0.1.1/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
 github.com/stretchr/testify v1.2.2/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXfy6kDkUVs=
 github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
-github.com/stretchr/testify v1.7.0 h1:nwc3DEeHmmLAfoZucVR881uASk0Mfjw8xYJ99tb5CcY=
 github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
+github.com/stretchr/testify v1.7.2 h1:4jaiDzPyXQvSd7D0EjG45355tLlV3VOECpq10pLC+8s=
+github.com/stretchr/testify v1.7.2/go.mod h1:R6va5+xMeoiuVRoj+gSkQ7d3FALtqAAGI1FQKckRals=
 github.com/subosito/gotenv v1.2.0/go.mod h1:N0PQaV/YGNqwC0u51sEeR/aUtSLEXKX9iv69rRypqCw=
 github.com/tmc/grpc-websocket-proxy v0.0.0-20190109142713-0ad062ec5ee5/go.mod h1:ncp9v5uamzpCO7NfCPTXjqaC+bZgJeR0sMTm6dMHP7U=
 github.com/xiang90/probing v0.0.0-20190116061207-43a291ad63a2/go.mod h1:UETIi67q53MR2AWcXfiuqkDkRtnGDLqkBTpCHuJHxtU=
@@ -321,8 +322,9 @@ gopkg.in/yaml.v2 v2.2.8/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
 gopkg.in/yaml.v2 v2.3.0 h1:clyUAQHOM3G0M3f5vQj7LuJrETvjVot3Z5el9nffUtU=
 gopkg.in/yaml.v2 v2.3.0/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
 gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
-gopkg.in/yaml.v3 v3.0.0-20200615113413-eeeca48fe776 h1:tQIYjPdBoyREyB9XMu+nnTclpTYkz2zFM+lzLJFO4gQ=
 gopkg.in/yaml.v3 v3.0.0-20200615113413-eeeca48fe776/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
+gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
+gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
 honnef.co/go/tools v0.0.0-20190102054323-c2f93a96b099/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=
 honnef.co/go/tools v0.0.0-20190106161140-3f1c8253044a/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=
 honnef.co/go/tools v0.0.0-20190418001031-e561f6794a2a/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=

gremlin-go/pom.xml

@@ -95,7 +95,36 @@ limitations under the License.
                                 </goals>
                                 <configuration>
                                     <skip>${skipTests}</skip>
-                                    <executable>./run.sh</executable>
+                                    <environmentVariables>
+                                        <GREMLIN_SERVER>${project.version}</GREMLIN_SERVER>
+                                        <!-- setting this env variable is needed to be cross-platform compatible -->
+                                        <HOME>${user.home}</HOME>
+                                    </environmentVariables>
+                                    <executable>docker-compose</executable>
+                                    <arguments>
+                                        <argument>up</argument>
+                                        <argument>--build</argument>
+                                        <argument>--exit-code-from</argument>
+                                        <argument>gremlin-go-integration-tests</argument>
+                                    </arguments>
+                                </configuration>
+                            </execution>
+                            <execution>
+                                <id>shutdown-container</id>
+                                <phase>integration-test</phase>
+                                <goals>
+                                    <goal>exec</goal>
+                                </goals>
+                                <configuration>
+                                    <skip>${skipTests}</skip>
+                                    <environmentVariables>
+                                        <GREMLIN_SERVER>${project.version}</GREMLIN_SERVER>
+                                        <HOME>${user.home}</HOME>
+                                    </environmentVariables>
+                                    <executable>docker-compose</executable>
+                                    <arguments>
+                                        <argument>down</argument>
+                                    </arguments>
                                 </configuration>
                             </execution>
                         </executions>