A Practical Guide to Querying Amazon Neptune Using Gremlin

Takahiro Iwasa

11/27/2021

6 min read

Graph Database Neptune

This note describes how to use Amazon Neptune.

What is a Graph Database

Graph databases are optimized for storing and querying relationships between entities. Unlike relational or NoSQL databases, graph databases excel in scenarios where relationships are complex and performance is critical.

In graph databases:

• Vertices represent entities.
• Edges define relationships between vertices.
• Both vertices and edges can have properties (key-value pairs), which is called a property graph.

Transactions in Neptune

Neptune manages read-only and mutation queries using distinct transaction isolation levels. Refer to the official documentation.

For read-only queries, Neptune uses snapshot isolation within MultiVersion Concurrency Control (MVCC). This ensures that dirty reads, non-repeatable reads, and phantom reads are prevented by using a snapshot taken at the start of the transaction.

For mutation queries, Neptune uses READ COMMITTED isolation to prevent dirty reads. Additionally, Neptune locks the record set being read, ensuring that non-repeatable reads and phantom reads are avoided.

Traversal Examples

Preparing Graph Data

This example uses a graph where vertices have an age property and edges define a weight property.

Use the following command to load the sample data provided above. The %%gremlin is a Jupyter Notebook magic command specifically designed for use with Neptune Workbench.

1
%%gremlin
2
// Drop existing data
3
g.V().drop()

1
%%gremlin
2
// Add Vertices
3
g.addV('person').property(id, 'A').property('age', 30)
4
 .addV('person').property(id, 'B').property('age', 25)
5
 .addV('person').property(id, 'C').property('age', 35)
6
 .addV('person').property(id, 'D').property('age', 20)
7
 .addV('person').property(id, 'E').property('age', 18)
8
 .addV('person').property(id, 'F').property('age', 25)
9
 .addV('person').property(id, 'G').property('age', 10)
10
 .addV('person').property(id, 'H').property('age', 15)

1
%%gremlin
2
// Add Edges
3
g.V('A').addE('know').to(g.V('B')).property('weight', 1.0)
4
 .V('A').addE('know').to(g.V('C')).property('weight', 0.9)
5
 .V('A').addE('know').to(g.V('H')).property('weight', 0.5)
6
 .V('B').addE('know').to(g.V('D')).property('weight', 0.8)
7
 .V('B').addE('know').to(g.V('E')).property('weight', 0.4)
8
 .V('C').addE('know').to(g.V('F')).property('weight', 0.5)
9
 .V('C').addE('know').to(g.V('G')).property('weight', 0.6)
10
 .V('D').addE('know').to(g.V('E')).property('weight', 0.7)
11
 .V('H').addE('know').to(g.V('E')).property('weight', 1.0)
12
 .V('H').addE('know').to(g.V('G')).property('weight', 1.0)

Example 1: Retrieving All Vertices

1
%%gremlin
2
// Extract Vertices
3
g.V()
4
 .project('id', 'properties') // Projection
5
 .by(id()).by(valueMap()) // valueMap returns properties of vertices.

Result:

Example 2: Traversing Connections

Retrieve all persons older than 25 and connected to 'A' within two levels:

1
%%gremlin
2
// Extract persons (entities) which are older than 25 years old and connected from A up to 2nd.
3
g.V('A')
4
 .repeat(outE().inV()).times(2).emit() // Repeat traversal of adjacent vertices twice
5
 .has('age', gte(25)) // Greater than or equal 25 years old
6
 .project('id', 'age')
7
 .by(id()).by(values('age'))

Result:

Example 3: Filtering by Weight

Find persons where the multiplied weight exceeds 0.5:

1
%%gremlin
2
// Start traversal at A which extracts vertices that have a multiplied weight value greater than 0.5
3
g.withSack(1.0f).V('A') // Sack can be used to store temporary data
4
 // Multiply a weight value of an out-directed edge by a sack value, and traverse all in-directed vertices
5
 .repeat(outE().sack(mult).by('weight').inV().simplePath()).emit()
6
 .where(sack().is(gt(0.5))) // A sack value greater than 0.5
7
 .dedup() // deduplication
8
 .project('id', 'weight')
9
 .by(id).by(sack())

Result:

Visualizing Graphs

Neptune Workbench offers tools to visualize graphs interactively. Refer to the official documentation for more details.

To visualize the graph, execute the following command:

1
%%gremlin -d T.id -de weight
2
// -d specifies the vertex property to display
3
// -de specifies the edge property to display
4

5
// Execute traversal from example 3
6
g.withSack(1.0f).V('A') // Sack is used to store temporary data
7
 .repeat(outE().sack(mult).by('weight').inV().simplePath()).emit() // Traverse with edge weight
8
 .where(sack().is(gt(0.5))) // Filter paths where the sack value > 0.5
9
 .dedup() // Remove duplicate paths
10
 .path() // Extract path data
11
 .by(elementMap()) // Display properties of vertices and edges

Example Output:

Once executed, navigate to the Graph tab in Neptune Workbench to visualize the result. The interface supports drag, zoom-in, and zoom-out operations, allowing for intuitive graph exploration.

Appendix 1: Tree Structures in Relational Databases (RDB)

Tree structures can be modeled in relational databases (RDB) using various approaches. In some scenarios, graph databases may not be necessary.

However, it is important to note that the Naive Tree model is often suboptimal for many cases, as discussed in “SQL Antipatterns”.

• Naive Tree
  • Expression: t1.id = t2.parent_id
  • Pros
    • Simple implementation (Adjacency List)
  • Cons
    • Difficult to extract non-adjacent nodes
    • Complex SQL
    • Low performance
• Path Enumeration
  • Expression: path LIKE '1/2%'
  • Pros
    • Simplifies extracting non-adjacent nodes
  • Cons
    • Complex INSERT/UPDATE/DELETE operations
    • Limited by column max length
• Nested Sets
  • Expression: Left > 1 AND Right < 6
  • Pros
    • Efficient for querying non-adjacent nodes
  • Cons
    • Complex INSERT/UPDATE operations
    • Limited by column max length
    • Non-intuitive structure
• Closure Table
  • Expression: Separate table for the tree
  • Pros
    • Efficient for querying all nodes
    • Handles INSERT/UPDATE/DELETE easily
  • Cons
    • Data can grow significantly in size
    • INSERT/UPDATE/DELETE can have low performance

When storing data in an RDB, querying relationships can become increasingly complex. SQL queries often grow in complexity, and representing interconnected data within a row-column structure is unintuitive.

Appendix 2: Transactions

Dirty Read

When Tx1 updates the row, Tx2 reads the row, and Tx1 fails or rollbacks, Tx2 reads data which has not been reflected.

Non-repeatable Read

When Tx1 reads the row, Tx2 updates or deletes the row and commits, and Tx1 reads the row again, Tx1 reads committed data different from the previous one.

Phantom Read

When Tx1 reads the records, Tx2 inserts or deletes some rows and commits, and Tx1 reads the rows again, Tx1 reads the rows different from the previous ones.

Isolation Levels