[Bug] Subset of subgraphs refuse to deploy, including all Network subgraphs

[gemma-lunanova]

Bug report

Four months ago we launched a new bare-metal indexing stack with Postgres 17. It is currently using graph-node version v0.36.1 and is successfully indexing and serving queries on over 1000 subgraphs across 16 networks. Unfortunately we have a pernicious issue where a small number of subgraphs across several different networks cannot be deployed.

Critically, all versions of the Network Subgraph are affected by this. Our legacy graph stack is still able to index the Network subgraph so we are currently having to maintain this in parallel but this is rapidly becoming untenable and we would greatly appreciate any help to resolve our issue.

Deploying problematic subgraphs using the indexer-agent or directly using graphman sees the same outcome: a row is created in the table subgraphs.subgraph but no new entries appear in any of the other subgraph tables. The subgraph appears in the offchain sync list in the CLI but doesn't appear at all in graphman info. Attempting to deploy again with graphman returns the message Error: Subgraph with name already exists

Subgraphs we encounter this issue with are:

QmUzRg2HHMpbgf6Q4VHKNDbtBEJnyp5JWCh2gUX9AV6jXv
QmaJSe4JD7wJvA5syMPxvvp8TNGY7GtBa9kUxALR8RYmhp
QmYBqjbjdbBmbnGvC1nRGpgEkUL14dw77d4L6PMcLPSu71
QmYH72PBbrmoeTdcg2SeQs2AYwWgxe538sxjzDqWT59MnF
QmZTk1nPtoWpepQRqD8UeAS5EXKfzKpnybhrotPvzbwok1
Qma4DUFeePzeprf9DfmbrZ1hVEoBBGVgqmsAB1NDQMw7kc
QmZxBnfcCqimGA8SVFfjbErXzh848WSoyuyitDmFPb9rXP
QmaTXJtSdYh1p9HoZtToYYF99zavdhsZKfJkbrLgsSUNmj
QmbxC2AYMvnTq6UBAvkAch6KUsRy3VaGHgt6s6c8nfa1hm
QmcpFsC6uE1ACPyfof1PgZgmT5Kpfh7UNtrDnbN5Q6qUuQ
QmdKXcBUHR3UyURqVRQHu1oV6VUkBrhi2vNvMx3bNDnUCc
Qmdy1ecqP39V1Z5XiBTYQRwSKkHwvUycpUfS2ACfofctq7
QmeB7YfNvLbM9AnSVeh5JvsfUwm1KVCtUDwaDLh5oxupGh
QmfVbF4NzrA1qn6PrefrzBHsYywT3k8tj7g3CQuGuysdTq
Qmf6cug5yWnEQLoBdr7trNTzFv93KmMBDzV553qZNfF1ML
Qmcnxu14tCGh3SQG5ktyXmXq8JHvDsge6j9419JJKMoZaH
QmPdhxTwAbWxUkQEZGz7yLUmiZuSommv59raCRjvYPN7k2
QmQ77fpqaa7fT8najCnU5dayunqNWASG3WMkfNxg93JF2y
QmRCMZJjxi3oKCtbh5CY4gBT2E21Cbz65DAzmnKMNPNGLV
QmRNDurj3yDRfmdf89Bv9MZASSr1fTSu9hdp3CgLTRY8rg
QmRUifGvSjKJNmki5tesMyX9QazRztyLTENnoH8zUrGnhY
QmRRKcjRN9BP51SEecq2AM1VXqjoS939ApE1mE4KtcZZtT
QmSVo91WL7RaB1oXUFhKAKCrd3bs8WjyownAb9QCK5MyGt
QmSctFnXhi1mA2qHofWXPTyQNLGRVCEKT5mgJYh591tRAL
QmTGxn3QEVCamxKC443V6KFqdSgKnwAZ59d4xsD4C7oFbe
QmTajxCVo8xXNAUoN553D5NqY3fMLZAeswkDKETY3EfFt9
QmPWDTWRfS8XewMJhxXoQiZX6RkrUMYH7BFnnB846v6wzR
QmVQeWqe6VKbwAugSYoA977xs1BdxeBL9Usn5LgBRRfXaK
QmVQg4uYVXN3stdxW338vXvcLFEp3QhyFeGutTB3c9VAoB
QmW4Vo3ZYV79pizzYKbNZ2TTKfHQhdRQDrmGDG25UkBpuz
QmW5vAS3VSGwRWuHFnCBsCfLMG1PJ6nxqSqjzWwhJwigQH
QmWJiduWv7RpLckjMEsXCKffwAcdAsM8gfnTfV642CwfMr
QmWg3CBE49J6Rjx5bbXRA6yown4gxd5eCPfyATfGRcoMZF
QmWpzXBkL6SgKiH3yNSnZA4qqd2tnU7vtoo35KTUqygPYP
QmTTXzn6iBcAX2Z72yJw6e64yfCX6c5HcTpKaRacfTLa7g
QmXtiD3cPJ51x4cyzCy51Cw8f8V64AwoejRWZyR8EtaGyo
QmY24oG6or5htCnAKfd1jdEn3LxtoRD8whHCmcQiapuhqb
QmXzb8m25vaAywjhGrzdR2KCjWLt5RGEcGKQ9BmvHEpaWY

In 'log output' are the errors/messages we see with the different deployment routes (via CLI using rules prepare and via graphman using deploy)

Deploying other subgraphs via either of these routes works as expected for all networks that we are currently supporting.

Relevant log output

**Deploying via the cli / agent:
**(graph indexer rules prepare returns usual output on the CLI)

**Agent logs:

Mar 20 16:44:49 <lunanova-indexer-agent>[133010]: {"level":30,"time":1742489089390,"pid":133010,"hostname":"graph-agent-red1","name":"IndexerAgent","component":"Agent","function":"reconcileDeployments","name":"indexer-agent/xALR8RYmhp","deployment":{"bytes32":"0xb1bb6f22826a21a64a306c5c6e2a4108cb39ec9cccf82a88884ee3d4a9f887a7","ipfsHash":"QmaJSe4JD7wJvA5syMPxvvp8TNGY7GtBa9kUxALR8RYmhp"},"msg":"Index subgraph deployment"}

Mar 20 16:44:49 <lunanova-indexer-agent>[133010]: {"level":20,"time":1742489089390,"pid":133010,"hostname":"graph-agent-red1","name":"IndexerAgent","component":"GraphNode","name":"indexer-agent/xALR8RYmhp","deployment":"QmaJSe4JD7wJvA5syMPxvvp8TNGY7GtBa9kUxALR8RYmhp","msg":"Ensure subgraph deployment is syncing"}

Mar 20 16:44:49 <lunanova-indexer-agent>[133010]: {"level":20,"time":1742489089409,"pid":133010,"hostname":"graph-agent-red1","name":"IndexerAgent","component":"GraphNode","name":"indexer-agent/xALR8RYmhp","deployment":"QmaJSe4JD7wJvA5syMPxvvp8TNGY7GtBa9kUxALR8RYmhp","msg":"Subgraph deployment not found, creating subgraph name and deploying..."}

Mar 20 16:44:49 <lunanova-indexer-agent>[133010]: {"level":30,"time":1742489089409,"pid":133010,"hostname":"graph-agent-red1","name":"IndexerAgent","component":"GraphNode","name":"indexer-agent/xALR8RYmhp","msg":"Create subgraph name"}

Mar 20 16:44:49 <lunanova-indexer-agent>[133010]: {"level":30,"time":1742489089411,"pid":133010,"hostname":"graph-agent-red1","name":"IndexerAgent","component":"GraphNode","name":"indexer-agent/xALR8RYmhp","msg":"Successfully created subgraph name"}

Mar 20 16:44:49 <lunanova-indexer-agent>[133010]: {"level":30,"time":1742489089411,"pid":133010,"hostname":"graph-agent-red1","name":"IndexerAgent","component":"GraphNode","name":"indexer-agent/xALR8RYmhp","deployment":{"bytes32":"0xb1bb6f22826a21a64a306c5c6e2a4108cb39ec9cccf82a88884ee3d4a9f887a7","ipfsHash":"QmaJSe4JD7wJvA5syMPxvvp8TNGY7GtBa9kUxALR8RYmhp"},"msg":"Deploy subgraph deployment"}

Mar 20 16:44:50 <lunanova-indexer-agent>[133010]: {"level":50,"time":1742489090012,"pid":133010,"hostname":"graph-agent-red1","name":"IndexerAgent","component":"GraphNode","name":"indexer-agent/xALR8RYmhp","deployment":

{"bytes32":"0xb1bb6f22826a21a64a306c5c6e2a4108cb39ec9cccf82a88884ee3d4a9f887a7","ipfsHash":"QmaJSe4JD7wJvA5syMPxvvp8TNGY7GtBa9kUxALR8RYmhp"},"err":

{"type":"IndexerError","message":"Failed to deploy subgraph deployment","stack":"IndexerError: Failed to deploy subgraph deployment\n    
at indexerError (/home/graph/indexer/packages/indexer-common/dist/errors.js:179:12)\n    at GraphNode.deploy (/home/graph/indexer/packages/indexer-common/dist/graph-node.js:276:51)\n    
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at async GraphNode.ensure (/home/graph/indexer/packages/indexer-common/dist/graph-node.js:390:17)\n    
at async /home/graph/indexer/packages/indexer-agent/dist/agent.js:553:13\n    
at async run (/home/graph/indexer/node_modules/p-queue/dist/index.js:163:29)","code":"IE026","explanation":"https://github.com/graphprotocol/indexer/blob/main/docs/errors.md#ie026","cause":{"type":"Error","message":"socket hang up","stack":"Error: socket hang up\n    
at connResetException (node:internal/errors:787:14)\n    
at Socket.socketOnEnd (node:_http_client:519:23)\n    
at Socket.emit (node:events:530:35)\n    
at Socket.emit (node:domain:488:12)\n    
at endReadableNT (node:internal/streams/readable:1696:12)\n    
at process.processTicksAndRejections (node:internal/process/task_queues:82:21)","code":"ECONNRESET"}},"msg":"Failed to deploy subgraph deployment"}

**Deploying via graphman:

:~$ graphman --config config.toml deploy -c -u http://<internal-ip>:port indexer-agent/xALR8RYmhp QmaJSe4JD7wJvA5syMPxvvp8TNGY7GtBa9kUxALR8RYmhp

Creating subgraph `indexer-agent/xALR8RYmhp`
Subgraph `indexer-agent/xALR8RYmhp` created
Deploying subgraph `QmaJSe4JD7wJvA5syMPxvvp8TNGY7GtBa9kUxALR8RYmhp` to `indexer-agent/xALR8RYmhp`
Error: Failed to deploy subgraph `QmaJSe4JD7wJvA5syMPxvvp8TNGY7GtBa9kUxALR8RYmhp` to `indexer-agent/xALR8RYmhp`

Caused by:
    0: error sending request for url (http://<internal-ip>:port/)
    1: client error (SendRequest)
    2: connection closed before message completed

IPFS hash

No response

Subgraph name or link to explorer

No response

Some information to help us out

• Tick this box if this bug is caused by a regression found in the latest release.
• Tick this box if this bug is specific to the hosted service.
• I have searched the issue tracker to make sure this issue is not a duplicate.

OS information

Linux

[mikenikles]

Our legacy graph stack is still able to index the Network subgraph

What version of graph-node does your legacy stack use? I see an increase in socket hang up errors too since upgrading to 0.36.1. However, I also made a change on the client that calls graph-node and I'm trying to narrow down if my client-side change or the graph-node upgrade causes the issue.

[gemma-lunanova]

Our legacy node is on 0.34.1…