Why recursive metadata calls are the wrong tool

Apps that need a complete view of a user’s Dropbox often reach for repeated /metadata calls: one per folder, each made over the network. For accounts with many directories, that adds up fast. The Core API’s /delta endpoint avoids the problem entirely. Although designed for incremental sync, the first call — with no cursor — returns a flat list describing every existing file and folder, replacing a whole tree of round trips with a single enumeration pass.

Turning delta entries into a local state

Handling /delta responses requires a little care. Each entry is an instruction, not a snapshot: attached metadata means the path should be added or updated in your local state, while a missing metadata block means the path should be removed (and, for folders, everything beneath it). Responses also include has_more and a new cursor; when has_more is true, you must immediately call /delta again with that cursor to fetch the next batch.

The Python function below keeps a dictionary of paths to metadata objects, applying deletions as prefixes so removing a folder clears its descendants, and loops until no more pages remain.

def list_files(client, files=None, cursor=None):
 if files is None:
 files = {}
 
 has_more = True
 
 while has_more:
 result = client.delta(cursor)
 cursor = result['cursor']
 has_more = result['has_more']
 
 for lowercase_path, metadata in result['entries']:
 
 if metadata is not None:
 files[lowercase_path] = metadata
 
 else:
 # no metadata indicates a deletion
 
 # remove if present
 files.pop(lowercase_path, None)
 
 # in case this was a directory, delete everything under it
 for other in files.keys():
 if other.startswith(lowercase_path + '/'):
 del files[other]
 
 return files, cursor

The function also accepts an existing dictionary and cursor, making it usable for ongoing updates rather than just a first-time walk. With a valid DropboxClient, you hand the result to a trivial sort-and-slice to surface the largest files:

from dropbox.client import DropboxClient
 
# ...
 
files = list_files(DropboxClient(token))
 
print 'Top 10 biggest files:'
 
for path, metadata in nlargest(10, files.items(), key=lambda x: x[1]['bytes']):
 print 't%s: %d bytes' % (path, metadata['bytes'])

Notes on the approach

  • The Sync and Datastore SDK (and its cached listFolder helpers) has been deprecated; this is for Core API clients that must talk to the network directly.
  • An access token is required to run the example, obtained either from existing code or by following the Core API Python tutorial.
  • Full runnable source is included here for reference.
import heapq
import sys
 
from dropbox.client import DropboxClient
 
if len(sys.argv) == 2:
 token = sys.argv[1]
else:
 print 'Usage: python app.py <access token>'
 sys.exit(1)
 
def list_files(client, files=None, cursor=None):
 if files is None:
 files = {}
 
 has_more = True
 
 while has_more:
 result = client.delta(cursor)
 cursor = result['cursor']
 has_more = result['has_more']
 
 for lowercase_path, metadata in result['entries']:
 
 if metadata is not None:
 files[lowercase_path] = metadata
 
 else:
 # no metadata indicates a deletion
 
 # remove if present
 files.pop(lowercase_path, None)
 
 # in case this was a directory, delete everything under it
 for other in files.keys():
 if other.startswith(lowercase_path + '/'):
 del files[other]
 
 return files, cursor
 
files, cursor = list_files(DropboxClient(token))
 
print 'Total Dropbox size: %d bytes' % sum([metadata['bytes'] for metadata in files.values()])
 
print
 
print 'Top 10 biggest files:'
 
for path, metadata in heapq.nlargest(10, files.items(), key=lambda x: x[1]['bytes']):
 print 't%s: %d bytes' % (path, metadata['bytes'])