ASP.NET Core Pagination For Large Datasets

Read time: 6 minutes

Today I’ll challenge what you know about pagination in ASP.NET Core.

You probably know how important it is to paginate your data before sending it from your API backend to your frontend.

However, most tutorials you’ll find on the Web follow an approach that is not efficient for large datasets, resulting in significant load for your database.

If you have to deal with large amounts of data, you need a different approach to pagination, and that’s what I’ll show you today.

Let’s dive in.

Pagination for small datasets

Most tutorials you’ll find on the Web will show you how to do what is known as offset pagination, which is an approach where you divide the total number of rows by the number of rows to display per page, and then you let the user choose the page number they want to see.

Image may be NSFW.
Clik here to view.

In the backend, the C# code use to retrieve the rows if also using Entity Framework Core and a relational database, would look something like this:

varpageNumber=3;varpageSize=5;vargames=dbContext.Games.OrderBy(game=>game.Id).Skip(pageNumber*pageSize).Take(pageSize);

The problem with this approach is that it’s not efficient. The database must still process the first 15 entries, even if they aren’t returned to the application.

This can result in significant load for your database that increases with the number of rows being skipped. So, it’s not a good approach for large datasets.

Let’s see what’s the recommended approach for large datasets.

Pagination for large datasets

Keyset pagination, also known as seek-based pagination, is a more efficient way to paginate through a large dataset. It’s based on the idea of using a unique key to fetch the next set of rows.

This means that you don’t need to skip any rows. You just need to remember the last key you fetched and use it to fetch the next set of rows.

Your C# LINQ query would now look something like this:

varlastId=5;varpageSize=5;vargames=dbContext.Games.OrderBy(game=>game.Id).Where(game=>game.Id>lastId).Take(pageSize);

As long as the Id column is indexed, this query will be very efficient, even for large datasets.

Let’s see how to implement keyset pagination in a full-stack ASP.NET Core application.

The DTOs

We will use 2 DTOs to represent the request and response for the pagination.

Here’s the DTO to represent the request:

publicrecordclassGetGamesDto(int?Cursor,// The last ID fetchedbool?IsNextPage,// A flag to indicate if we want the next page or the previous pageintPageSize=5);// The number of rows to fetch

And this is the DTO to represent the response:

publicrecordclassGamesPageDto(IEnumerable<GameSummaryDto>Data,// The list of resourcesint?NextId,// The next ID to fetchint?PreviousId,// The previous ID to fetchboolIsFirstPage);// A flag to indicate if this is the first page

For completeness, here’s the DTO to represent the game summary, although it’s not really relevant to the pagination:

publicrecordclassGameSummaryDto(intId,stringName,decimalPrice);

Now, let’s see how to implement the pagination logic in the backend.

Backend API implementation

I’ll show you an implementation that lets you fetch both the next and previous pages, which can be a bit tricky.

If you only need the next page (common in infinite scrolling scenarios) the logic would be simpler.

The first step is to order your rows by the Id column, to ensure that the rows are returned in the same order every time, and to take advantage of the index:

app.MapGet("/games",async(CatalogContextdbContext,[AsParameters]GetGamesDtorequest)=>{// Order by Id for keyset paginationIQueryable<Game>games=dbContext.Games.OrderBy(game=>game.Id);});

Then, we need to select the rows based on the Cursor and IsNextPage parameters:

// Take 1 extra record to check if there's a next page.inttakeAmount=request.PageSize+1;if(request.Cursorisnotnull){if(request.IsNextPage==true){// Fetch the next pagegames=games.Where(game=>game.Id>request.Cursor);}else{// Fetch the previous pagegames=games.Where(game=>game.Id<request.Cursor).OrderByDescending(game=>game.Id);// No extra record needed in this casetakeAmount=request.PageSize;}}games=games.Take(takeAmount);// Reverse the list if it's a previous page requestif(request.IsNextPage==false&&request.Cursorisnotnull){games=games.Reverse();}

Notice how we are not skipping any rows. We are just filtering the rows based on the Id column and wether we want the next or previous page.

Now that the games variable contains the rows we want to return, let’s make sure we include the game genres, convert the rows to the DTOs, and turn everything into a list:

vargamesOnPage=awaitgames.Include(game=>game.Genre).Select(game=>game.ToGameSummaryDto()).AsNoTracking().ToListAsync();

Next, we need to do a few calculations to determine the NextId and PreviousId we’ll include in the response:

boolisFirstPage=!request.Cursor.HasValue||(request.Cursor.HasValue&&gamesOnPage.First().Id==dbContext.Games.OrderBy(g=>g.Id).First().Id);// There's a next page if:// 1. We got an extra record// 2. We're navigating to the previous pageboolhasNextPage=gamesOnPage.Count>request.PageSize||(request.Cursorisnotnull&&request.IsNextPage==false);// Remove the extra record used for next page detectionif(gamesOnPage.Count>request.PageSize){gamesOnPage.RemoveAt(gamesOnPage.Count-1);}int?nextId=hasNextPage?gamesOnPage.Last().Id:null;int?previousId=gamesOnPage.Count>0&&!isFirstPage?gamesOnPage.First().Id:null;

Finally, we return the response:

returnnewGamesPageDto(gamesOnPage,nextId,previousId,isFirstPage);

Like I said, it’s a bit tricky, and took me a while to get it right, but it’s a very efficient way to paginate through large datasets.

Now let’s see how to implement the frontend.

Blazor frontend implementation

In our Blazor Static SSR application, we’ll start by defining a typed client that can make use of the HttpClient to make requests with the expected parameters to the backend:

publicclassGamesClient(HttpClienthttpClient){publicasyncTask<GamesPage>GetGamesAsync(int?cursor,boolisNextPage,intpageSize){varquery=QueryString.Create("pageSize",pageSize.ToString()).Add("isNextPage",isNextPage.ToString());if(cursorisnotnull){query=query.Add("cursor",cursor.Value.ToString());}returnawaithttpClient.GetFromJsonAsync<GamesPage>($"games{query}")??newGamesPage([],null,null,true);}}

To understand how that GamesClient instance is registered, checkout my HTTP Client Tutorial.

For completeness, here’s the GamesPage record, which is mostly a copy of the GamesPageDto class used in the backend:

publicrecordclassGamesPage(IEnumerable<GameSummary>Data,int?NextId,int?PreviousId,boolIsFirstPage);

We will also need to implement a record that we’ll call PaginationInfo, which will be handy in our upcoming Pagination component:

publicrecordclassPaginationInfo(int?NextId,int?PreviousId,boolIsFirstPage){publicboolHasPrevious=>!IsFirstPage&&PreviousIdisnotnull;publicboolHasNext=>NextIdisnotnull;}

Let’s now create the actual Pagination.razor component, which will be in charge of rendering our Previous and Next links, based on the information we get from the PaginationInfo record:

@injectNavigationManagerNavigation@if(PaginationInfoisnotnull){<nav><ulclass="paginationjustify-content-center">
<liclass="page-item@(!PaginationInfo.HasPrevious?"disabled":null)">
<aclass="page-link" href="@PaginationUri(PaginationInfo.PreviousId,false)">
Previous</a></li><liclass="page-item@(!PaginationInfo.HasNext?"disabled":null)">
<aclass="page-link" href="@PaginationUri(PaginationInfo.NextId,true)">
Next</a></li></ul></nav>}@code{[Parameter]publicPaginationInfo?PaginationInfo{get;set;}privatestringPaginationUri(int?cursor,boolisNextPage)=>Navigation.GetUriWithQueryParameters(newDictionary<string,object?>(){{"cursor",cursor},{"isNextPage",isNextPage}});}

Finally, we can use both our GamesClient and Pagination components in our Home.razor component:

@page"/"@injectGamesClientClient@attribute[StreamRendering]<PageTitle>GameStore</PageTitle>@if(gamesPageisnull||paginationInfoisnull){<pclass="mt-3"><em>Loading...</em></p>
}else{<divclass="rowrow-cols-1row-cols-md-5mt-3">
@foreach(vargameingamesPage.Data){<divclass="col">
<ahref="game/@game.Id"style="text-decoration: none;"><divclass="cardh-100">
<divclass="card-img-container">
<imgclass="card-img-top" src="@game.ImageUri">
</div><divclass="card-body">
<h5class="card-title">@game.Name</h5>
<pclass="card-text">@game.Price.ToString("C2")</p>
</div></div></a></div>}</div><divclass="rowmt-2">
<divclass="col">
<PaginationPaginationInfo="paginationInfo"/></div></div>}@code{privateGamesPage?gamesPage;PaginationInfo?paginationInfo;constintPageSize=5;[SupplyParameterFromQuery]publicint?Cursor{get;set;}[SupplyParameterFromQuery]publicbool?IsNextPage{get;set;}protectedoverrideasyncTaskOnInitializedAsync(){gamesPage=awaitClient.GetGamesAsync(Cursor,IsNextPage??false,PageSize);paginationInfo=newPaginationInfo(gamesPage.NextId,gamesPage.PreviousId,gamesPage.IsFirstPage);}}

The end result

Here’s a screenshot of the full-stack ASP.NET core application, with keyset pagination enabled:

Image may be NSFW.
Clik here to view.

Notice the urls produced by the Pagination component, which include the cursor and isNextPage parameters.

Mission accomplished!

Whenever you’re ready, there are 3 ways I can help you:

1. ​Building Microservices With .NET:​ The only .NET backend development training program that you need to become a Senior .NET Backend Engineer.

2. ASP.NET Core Full Stack Bundle: A carefully crafted package to kickstart your career as an ASP.NET Core Full Stack Developer, step by step.

3. Promote yourself to 15,000+ subscribers by sponsoring this newsletter.