Open in app Sign up Sign in

Search Write

How To Do Pagination in Postgres

with Golang in 4 Common Ways
A few examples of pagination on Postgres, with Benchmark, written
on Golang

Iman Tumorang · Follow

Published in Easyread · 9 min read · May 26, 2020

568 10

Sign up to discover human stories that deepen your understanding

of the world.

Membership

Free
Access the best member-only stories.

Distraction-free reading. No ads. Support independent authors.

Organize your knowledge with lists and Listen to audio narrations.

highlights.
Read offline.
Tell your story. Find your audience.
Join the Partner Program and earn for
your writing.

Sign up for free

Try for $5/month

Sign up to discover human stories that deepen your understanding
Photo by Ergita Sela on Unsplash

of the world.
Hi again everyone, it’s been a long time since I haven’t published any article.
There are a lot of things happening, like the pandemic and much more stuff.
This pandemic is affecting me mentally like this self-quarantine is
Membership
exhausting
Free and stressing me. I hope this Covid-19 pandemic will be ended
Access the best member-only stories.
before Christmas this year. 😭
Distraction-free reading. No ads. Support independent authors.

On this rare your

Organize occasion, after
knowledge fighting
with lists and boredom and
Listenlaziness, I found the spirit
to audio narrations.
highlights.
to finish this article. Starting from me when building our new application in
Read offline.
Tell your story. Find your audience.
my current job, I’m curious about a few things,Join
and in this part, it’s about
the Partner Program and earn for
pagination. Like how to handle pagination in ayour writing.
better way based on my
understanding,
SignLOL. *My proposed idea might not be the best, so if you guys have
up for free

a better way, a vast experience than me, put your comments below
Try for yaa!!
$5/month
TBH, I never cared more details about this in my previous job because we all
had the same perspective, and we only like to have 10 engineers in my
previous company, so we could have the same perspective. But now I care
about this since we have a lot of engineers in my current job, and everyone
has a different perspective.

So, I’m just curious, what’s the better way to build pagination on Postgres on
top of the application, with my case, I’m using Golang for the application.

Actually, there are two famous styles of pagination:

Cursor based pagination

Offset based pagination

Sign
In thisup to discover
article, I’ll only human stories
cover those thatpaginations
two style deepen yourin 4 understanding
different
common ways that Backend engineer of the usually
world.does, or at least what I know so
far since I know how to code.

Membership
Do pagination with page number, pretty common; the user only sends
Free
the page number, and we handle it internally; I use offset at the database
Access the best member-only stories.
level.
Distraction-free reading. No ads. Support independent authors.
Do pagination with offset and limit, pretty common since the RDBMS
Organize your knowledge with lists and Listen to audio narrations.
features.
highlights.The user will directly send the offset number from the query
Read offline.
param.
Tell your story. Find your audience.
Join the Partner Program and earn for
Do pagination with a simple query with an your
autowriting.
incremental ID as the PK,
Sign up for
quite common forfree
auto incremental ID in the database. Which is the ID is
treated as the cursor. Try for $5/month
 Do pagination with UUID as the PK combined with the created
timestamp, also known as the seek-pagination method or keyset
pagination method. And the combined key will be hashed into a cursor
string.

So what I’m gonna do here is I’ll create those four pagination

implementations and do a small benchmark from code; I’ll use Golang
Benchmark. The goal of this article is just to satisfy my curiosity LOL. I know
I can read people’s articles, but I want to do it with my version.

TL;DR

All the code used here has already been pushed to my Github repository,
github.com/bxcodec/go-postgres-pagination-example
SignConclusions
up to discover
can behuman stories
seen at the that
bottom deepen
of this your understanding
article
of the world.

Membership
Pagination On REST API
Free
Access the best member-only stories.
To give you some context, *in case you don’t know what pagination is used for.
Distraction-free
Pagination is used reading. No ads. your response, LOL.
to paginate Support independent
Well, I don’tauthors.
know how to
rephrase it better.
Organize your knowledge with lists and Listen to audio narrations.
highlights.
Read offline.
Tell your story. Find your audience.
I’ll create an example; let’s say I have this endpoint in REST API.
Join the Partner Program and earn for
your writing.

Sign up for free

GET /payments Try for $5/month

And this endpoint will fetch all payments from the API. As we know, in
bigger-scale applications with tons of data sets, these payments may have
thousands or millions of data rows. And as a user, I want to fetch my
payments list.

From a database perspective, querying all the records will takes time a lot. I
can imagine how long it will be if we have a million records and fetch all the
data. So, in that case, people introduce what they called pagination. It works
like pages on the books, that each page contains a bunch of words.

But for this endpoint, each page will contain a list of payment details, so we
can still fetch the payment faster but maybe it will truncated into multiple
pages until we can fetch all the payment records.

Sign up to discover human stories that deepen your understanding

GET /payments?page=1 // to fetch payments in page 1
of thepayments
GET /payments?page=2 // to fetch world. in page 2
GET /payments?page=3 // to fetch payments in page 3
... etc

Membership
YouFree
may have seen this style in any endpoint, or maybe something like this
Access the best member-only stories.
as well.
Distraction-free reading. No ads. Support independent authors.

Organize your knowledge with lists and Listen to audio narrations.

GEThighlights.
/payments?limit=10 // initial request for fetch payment
Read offline.
GET /payments?limit=10&cursor=randomCursorString // with cursor
Tell your story. Find your audience.
GET /payments?limit=10&cursor=newrandomCursorString // for next page
Join the Partner Program and earn for
GET /payments?limit=10&cursor=anotherNewrandomCursorString
your writing.
... etc
Sign up for free

Try for $5/month

And many more, this is what we called pagination. We truncate our list of
data into a few segments and send it to the client, so we still maintain the
performance of the application and the client won’t lose track when fetching
our data.

1. Pagination with Page Number

GET /payments?page=1 // to fetch payments in page 1

GET /payments?page=2 // to fetch payments in page 2
GET /payments?page=3 // to fetch payments in page 3
... etc

Have you seen pagination like those above? TBH, I never have seen any
pagination like those, not in public API if I remember correctly. But, I’ve ever
created pagination with that’s style, around 4 years ago, on my first job-test
after graduated.

Sign up to discover human stories that deepen your understanding

So the logic quite complicated in the backend, but it will simplify from the
of the world.
user experience,

First I’ll set the default limit, let’s say 10. Per page is 10 items.
Membership
And each page number will be multiplied to the default limit
Free
Access the best member-only stories.
Then I’ll use it as the offset to the database.
Distraction-free reading. No ads. Support independent authors.

And, the your

Organize user can fetch
knowledge theand
with lists items based on Listen
the requested page number.
to audio narrations.
highlights.
Read offline.
Tell your
So then, story.
I try to Find youragain
build audience.
a simple application for this kind of method.
Join the Partner Program and earn for
With 100K rows of data, I try to benchmark it. your writing.
Sign up for free

Benchmark Result Try for $5/month

Sign up to discover human stories that deepen your understanding
of the world.

Membership

Free
Access the best member-only stories.

Distraction-free reading. No ads.

Benchmark Support
Result using Go for PageNumber independent authors.
pagination

Organize your knowledge with lists and Listen to audio narrations.

highlights.
The drawback of this pagination method is Read offline.
Tell your story. Find your audience.
Join the Partner Program and earn for
Performance-wise, it’s not your writing.
recommended. The bigger the data set, the
bigger theSign up for freeconsumption.
resource
Try for $5/month

But the benefit of using this method, the user feels like opening a book, they
will just need to pass the page number.
2. Pagination with Offset and Limit
Pagination with offset and limit is quite common to engineers. This comes
because of the feature of RDBMS that supports offset and limit for querying.

From the application level, there’s no extra logic, just passing the offset and
limit to the database, and let the database do the pagination.

How is usually looks like,

GET /payments?limit=10 // initial

GET /payments?limit=10&offset=10 //fetch the next 10 items
GET /payments?limit=10&offset=20 //fetch the next 10 items again
Sign up to discover human stories that deepen your understanding
... etc
of the world.
From the client-side, they only need to add the offset params, and the API
will return the items based on the given offset.Membership

Free
Access
And from database level, which is RDBMS, it will thelike
look best member-only
this below, stories.

Distraction-free reading. No ads. Support independent authors.

Organize your knowledge with lists and Listen to audio narrations.

highlights.
SELECT
* Read offline.
Tell your story. Find your audience.
FROM
payments Join the Partner Program and earn for
ORDER BY created_time your writing.
LIMIT 10
Sign up for free
OFFSET 20;
Try for $5/month

Benchmark Result
Sign up to discover human stories that deepen your understanding
of the world.

Membership

Free
Access the best member-only stories.

Distraction-free reading. No ads.Result using Go for LimitOffset

Benchmark Support independent authors.
pagination

Organize your knowledge with lists and Listen to audio narrations.

highlights.
The drawback of this pagination method Read offline.
Tell your story. Find your audience.
Join the Partner Program and earn for
Performance-wise, it’s not your writing.
recommended. The bigger the data set, the
bigger theSign up for freeconsumption.
resource
Try for $5/month

The benefits of this pagination method

 Very easy to implement; no need to do complex logic things in the server

3. Pagination with Auto Incremental PK of the ID

This pagination method was also pretty common. We set our table to auto
increment, using that as the page identifier/cursor.

How it’s used in REST

GET /payments?limit=10
GET /payments?limit=10&cursor=last_id_from_previous_fetch
GET /payments?limit=10&cursor=last_id_from_previous_fetch
Sign up to discover human stories that deepen your understanding
... etc

of the world.
How it looks like in database query level

Membership

Free
SELECT Access the best member-only stories.
*
FROMDistraction-free reading. No ads. Support independent authors.
payments
WHERE
Organize your knowledge with lists and Listen to audio narrations.
Id > 10
highlights.
LIMIT 20 Read offline.
Tell your story. Find your audience.
Join the Partner Program and earn for
your writing.
Or for descending
Sign up for free

Try for $5/month

SELECT
*
 FROM
payments
WHERE
Id < 100
ORDER BY Id DESC
LIMIT 20

Benchmark Result

Sign up to discover human stories that deepen your understanding

of the world.

Membership

Free
Access the best member-only stories.

Distraction-free reading. No ads. Support independent authors.

Organize your knowledge with lists and Listen to audio narrations.

highlights.
Read offline.
Tell your story. Find your audience.
Join the Partner Program and earn for
your writing.

Sign up for free

Try for $5/month

Benchmark Result using Go for AutoIncrement pagination

The drawback of this pagination method

The only drawback of this pagination method is, that when using the
auto-increment id, it will be problematic in the world of microservice
and distributed systems.
Like id with 20 can exist in Service Payment and Service User. It’s unique
in the same application context. It will be different if each ID uses UUID,
it’s “practically unique” (which means there’s a very small possibility of
duplicate generated UUID). So some people are trying to use UUID
instead of PK. Read more details about UUID and auto-increment keys
here

The benefits of this pagination method

SignEasy
uptotoimplement, no need to do complex logic things in the server.
discover human stories that deepen your understanding
of that
The best way to do pagination the Iworld.
know so far performance-wise since
it’s using autoincrement ID.

Membership

Free
Access the best member-only stories.

4. Pagination with
Distraction-free UUID
reading. NoCombined
ads. with Created Timestamp
Support independent authors.

Organize
I’m not sure your
thisknowledge
is prettywith lists and
common, Listen
but I see that to audio
a few narrations.
articles do this kind
highlights.
of pagination. The context is that the table is not using
Read offline.auto incremental id,
Tell your story. Find your audience.
but it’s using the UUID instead. But then peopleJoin
wonder how
the Partner to do
Program and earn for
your writing.
pagination, adding a new column with auto incremental number is a wasting
resource. So Sign
for up for free what I do is, use the created timestamp of my rows
myself,
and combine it with the PK, which is the UUID. Try for $5/month
This is the database schema

Sign up to discover human stories that deepen your understanding

of the world.

Membership

Free
Access the best member-only stories.

Distraction-free reading. No ads. Support independent authors.

Organize your knowledge with lists and Listen to audio narrations.

highlights.
Read offline.
payment.sql schema with UUID and timestamp
Tell your story. Find your audience.
Join the Partner Program and earn for
your writing.
And for the faster queries, I make an index with multiple tables, which is the
Sign up for free
PK and the created timestamp; as you can see from the above schema, I
Try for $5/month
made an index named idx_payment_pagination .
So the logic is,

I’ll use the UUID, which is my primary key and combine it with create

timestamp

Combine those two into a string, then I encode it to a base64 string

And return that encoded string as a cursor for the next page, so the user
can use it to fetch the next page of their request.

Example of how I made the cursor on the application level

Sign up to discover human stories that deepen your understanding

of the world.

Membership

Free
Access the best member-only stories.

Distraction-free reading. No ads. Support independent authors.

Organize your knowledge with lists and Listen to audio narrations.

highlights.
Read offline.
Tell your story. Find your audience.
Join the Partner Program and earn for
your writing.

Sign up for free

Try for $5/month

Sign up to discover human stories that deepen your understanding
of the world.

Membership

Free
Access the best member-only stories.

Distraction-free reading. No ads. Support independent authors.

Organize
And this your knowledge
is what it lookswith
likelists
inand Listen to audio narrations.
the REST endpoint.
highlights.
Read offline.
Tell your story. Find your audience.
Join the Partner Program and earn for
GET /payments?limit=10 your writing.
GET /payments?limit=10&cursor=base64_string_from_previous_result
GET Sign up for free
/payments?limit=10&cursor=base64_string_from_previous_result
... etc
Try for $5/month
But in the database, the query will look like this,

SELECT *
FROM payments
WHERE created_time <= '2020-05-16 03:15:06' // created timestamp
AND id < '2a1aa856-ad26-4760-9bd9-b2fe1c1ca5aa' // this is UUID
ORDER BY created_time DESC, id DESC
LIMIT 2

Benchmark Result

Sign up to discover human stories that deepen your understanding

of the world.

Membership

Free
Access the best member-only stories.

Distraction-free reading. No ads. Support independent authors.

Organize your knowledge with lists and Listen to audio narrations.

highlights.
Read offline.
Tell your story. Find your audience.
Join the Partner Program and earn for
your writing.

Sign up for free

Try for $5/month

Sign up to discover human stories that deepen your understanding
of the world.

Membership

Free
Access the best member-only stories.

Distraction-free reading. No ads. Result using Go for Keyset

Benchmark Support independent authors.
pagination

Organize your knowledge with lists and Listen to audio narrations.

highlights.
The drawback of this pagination method Read offline.
Tell your story. Find your audience.
Join the Partner Program and earn for
The performance may not be the best like your writing.
using the autoincrement id.
Sign up for free
But it’s consistent even though we will have millions of data
Try for $5/month
Quite tricky and advanced, we need to understand the index because if
we didn’t add an index, this query would really take time on a big dataset.
 And also, we need to be careful when handling timestamps. Even I still
facing some issues when querying the timestamp when doing this.

The benefits of this pagination method

The ID is UUID, so it’s practically globally unique across microservice in

the organizations.

The performance is consistent from the beginning until querying the last
page of the data

Conclusions
Sign upafter
Alright, to discover
doing all human stories that
the benchmarks, deepen
I’ve come yourconclusions.
to some understanding
of the world.
1. Performances: Faster to Slower
From the benchmark results (using the Golang benchmark tool), the faster
one is using the autoincrement PK. See the chart below; the smaller, the
Membership
faster the chart for the average time needed for each operation in
Free
nanoseconds. This chart is might not be a goodAccess the best member-only
representation, stories.be
it should
betterDistraction-free
if I make itreading.
in 95th, 97th, etc percentile, Support
No ads. but I got this value
independent from the
authors.

benchmark
Organize result. So I assume
your knowledge this is already good
with lists and enough
Listen to for the
audio narrations.
highlights.
representation. Read offline.
Tell your story. Find your audience.
Join the Partner Program and earn for
your writing.

Sign up for free

Try for $5/month

Sign up to discover human stories that deepen your understanding
of the world.

Membership

Free
Access the best member-only stories.

Distraction-free reading. No ads. Support independent authors.

Organizewith
Pagination your knowledge with lists and
autoincrement Listen to audio
ID is faster, followed narrations.
by UUID/created time,
highlights.
and PageNumber and LimitOffset. And this is only with 100K rows of data.
Read offline.
Tell your story. Find your audience.
And it will grow bigger as the data grow as well. Sothewith
Join only
Partner 100K
Program anddata, even
earn for
your writing.
if it is still under 1 second, the differences are already quite high when using
Signcompared
autoincrement up for free to limit offset.
Try for $5/month
2. Development: Faster to Slower
Implementation difficulties from easy to hard
 Using Offset, because we just only pass the offset and limit directly to the
database.

Using PageNumber, this is opinionated; some people may have different

logic, but for my case, I put this in the top two.

Using autoincrement ID

Using UUID with created time

Code artifacts
For the code, I’ve pushed it to my GitHub repository, which can be found
here, https://github.com/bxcodec/go-postgres-pagination-example

The issues that I face when doing this

When doing all of these things, obviously I face some issues, but I’ve
resolved them, and I also learn about them. I’ve written the things that I
Sign up to discover human stories that deepen your understanding
learned here, as well in this article: TIL: Becareful on Postgres Query, for
of the world.
Less than Or Equal on Timestamp

Author Suggestion
Membership
As a software engineer and as the author of this article, I recommend using
Free
autoincrement ID when doing pagination, but Access
if yourthesystem or you stories.
best member-only don’t
want to use autoincrement ID as the PK, you may
Distraction-free reading. No ads.
consider using keyset
Support independent authors.
pagination; in my case using UUID + created_time timestamp.
Organize your knowledge with lists and Listen to audio narrations.
highlights.
Read offline.
Tell your story. Find your audience.
Join the Partner Program and earn for
your writing.

Reference Sign up for free

1. Tons of Stackoverflow answers, I forgot which one,
Trybut all answers that
for $5/month

related to pagination with Postgres.

 2. Faster SQL Pagination with jOOQ Using the Seek Method

3. REST API Design: Filtering, Sorting, and Pagination

Postgres Golang Pagination Database Software Engineering

Sign up to discover human stories that deepen your understanding

of the world.
Written by Iman Tumorang Follow

2.6K Followers · Editor for Easyread

Membership
Software Architect @ Xendit | Independent Software Architect @ SoftwareArchitect.ID |
Reach me at https://imantumorang.com for fast response :)
Free
Access the best member-only stories.

Distraction-free reading. No ads. Support independent authors.

Organize
More from Imanyour knowledge
Tumorang andwith lists and
Easyread Listen to audio narrations.
highlights.
Read offline.
Tell your story. Find your audience.
Join the Partner Program and earn for
your writing.

Sign up for free

Try for $5/month

 Iman Tumorang in GolangID Bismo Baruno in Easyread

Mencoba Clean Architecture pada Unit Test (SQL) in Golang

Golang How to mock our SQL code in Golang for Unit
Independent , Testable, and Clean Test

Oct 20, 2017 548 4 Jun 19, 2020 309 2

Ferdina Kusumah in Easyread Iman Tumorang in Easyread

Just Call Your Code Only Once !! Mount file to Kubernetes Pod
Sign upPattern
Singleton to discover
in Go human stories that deepen
Without yourtheunderstanding
Deleting Existing File
of the world.
What I learned about Kubernetes when trying
to mount config maps into a docker containe…

Jun 23, 2020 81 2 Jan 28, 2019 161 3

Membership

Free
Access the best member-only stories.

See all from Iman Tumorang

Distraction-free See all from Easyread
reading. No ads. Support independent authors.

Organize your knowledge with lists and Listen to audio narrations.

highlights.
Read offline.
Tell your story. Find your audience.
Join the Partner Program and earn for
your writing.

Sign up for free

Try for $5/month

Recommended from Medium
 Wahyu Hutomo Adji RimonTawadrous in ITNEXT

Basic Tutorial GORM Why UUID7 is better than UUID4 as

GORM is An object-relational mapper (ORM) clustered index in RDBMS
code library that automates the transfer of… In the Introduction To Database Indexing
Article, We discussed database indexes, The…

Jan 4 22 Jan 15 1.2K 13

Sign
Lists up to discover human stories that deepen your understanding
of the world.
General Coding Knowledge Stories to Help You Grow as a
20 stories · 1235 saves Software Developer
19 stories · 1072 saves
Membership
Leadership Good Product Thinking
Free 50 stories · 336 saves 11 stories · 582 saves
Access the best member-only stories.

Distraction-free reading. No ads. Support independent authors.

Organize your knowledge with lists and Listen to audio narrations.

highlights.
Read offline.
Tell your story. Find your audience.
Join the Partner Program and earn for
your writing.

Sign up for free

Try for $5/month

Yota Hamada Abdul Saleem Mohamed Faheem Anver

Mastering DDD: Repository Design Handling Race Conditions and
Patterns in Go Concurrent Resource Updates in…
This article aims to deepen the understanding // CPU-bound tasks that might impact
of “Repository”, a core concept in Domain-… concurrency due to Node running in a single…

Jan 2 273 5 Dec 21, 2023 5

Eugene Nikolaev Ruangyot Nanchiang

Data Pagination Using How to implement clean

Elasticsearch in Golang architecture in Golang (EN)
Sign up to discover human stories that
Elasticsearch, a powerful search and analytics
deepen your understanding
I have started to journey with my backend
engine, provides robust capabilities forof the world.
data… developer career for 1 year and 8 months. I…

Dec 25, 2023 Dec 6, 2023 451 12

Membership

Free
Access the best member-only stories.
See more recommendations
Distraction-free reading. No ads. Support independent authors.

Organize your knowledge with lists and Listen to audio narrations.

highlights.
Read offline.
Tell your story. Find your audience.
Join the Partner Program and earn for
your writing.

Sign up for free

Try for $5/month