A Different Kind of API App

Years ago, Linux kernel hacker Robert Love wrote beaglefs, a FUSE file system that populated user-created folders with Beagle search results, using the folder name as the query. The idea was to show the power of user-space file systems and the library code available outside the kernel. We can build something similar with the Dropbox API: a hypothetical app that fills user-created folders with Creative Commons–licensed images, found by searching the web with the folder name as the search term. To run an image search, the user just creates a folder.

The implementation in this article is written in Haskell, using the Haskell Platform (based on GHC, the Glasgow Haskell Compiler). The code is presented in literate Haskell style: lines prefixed with > are part of the actual program. The full source is available on GitHub.

Language Extensions and Modules

Haskell is over two decades old but still evolving. You enable newer language features with the LANGUAGE pragma, telling the compiler to allow specific extensions. The exact extensions used here aren’t important upfront; the GHC docs explain each one.

> {-# LANGUAGE TypeFamilies #-}
> {-# LANGUAGE QuasiQuotes #-}
> {-# LANGUAGE MultiParamTypeClasses #-}
> {-# LANGUAGE FlexibleContexts #-}
> {-# LANGUAGE TemplateHaskell #-}
> {-# LANGUAGE DeriveDataTypeable #-}

Imports bring definitions from other modules into the current namespace. For example, importing all definitions from the Yesod module, a full-featured web framework for Haskell, gives us the tools to build the app's web interface. Alternatively, the hiding syntax excludes specific names, as when we avoid importing when from HXT, the XML library used later to parse image search results.

> import Yesod
> import Text.XML.HXT.Core hiding (when)

Some imports are selective, bringing in only named definitions, similar to Python's from module import name. Adding qualified forces prefixing with the module name — for instance, as C lets us write C.try rather than the full Control.Exception.Lifted.try.

> import Control.Applicative ((<$>), (<*>))
> import Control.Concurrent (forkIO, threadDelay, newChan,
>                            writeChan, readChan, Chan, isEmptyChan)
> import Control.Monad (forM_, when)
> import Control.Monad.Base (MonadBase)
> import Control.Monad.Trans.Control (MonadBaseControl)
> import Data.Maybe (fromJust, mapMaybe, isNothing)
> import Data.Text.Encoding (decodeUtf8With)
> import Data.Text.Encoding.Error (ignore)
> import Data.Typeable (Typeable)
> import Network.HTTP.Enumerator (simpleHttp)
> import System.FilePath.Posix (takeFileName, combine)

> import qualified Control.Exception.Lifted as C
> import qualified Data.ByteString as B
> import qualified Data.ByteString.Lazy as L
> import qualified Data.List as DL
> import qualified Data.Map as Map
> import qualified Data.Text as T
> import qualified Data.URLEncoded as URL
> import qualified Dropbox as DB
> import qualified Network.URI as URI

Defining New Types

An algebraic datatype declaration is the basic way to define new types in Haskell. Though short, the declaration for ImageSearch does two things: it defines the type ImageSearch itself, and it defines a constructor — a function that takes two arguments and returns a value of type ImageSearch.

> data ImageSearch = ImageSearch (Chan (DB.AccessToken, String)) DB.Config

Constructors are central to Haskell. Wherever a name can be bound to a value, you can use constructor pattern matching (also called deconstruction) to extract the contained data. For example, a function to get the channel component from our type would pattern-match on the constructor and return the first field. Type names and value names live in different namespaces, so it's fine that the type and its constructor share the name.

The Maybe Type

The Maybe type is an algebraic datatype you'll encounter frequently. It's used to denote an error value from a function, or an optional argument. Unlike ImageSearch, Maybe has two constructors: Nothing (usually an error) and Just (indicating success).

data Maybe a = Nothing | Just a

But Maybe itself isn't a concrete type — it has to wrap some other type. This higher-order type, called a type constructor, takes a concrete type and returns a new one, like Maybe Int or Maybe String. This is similar to generics like List<T> in Java or C#. The type variable a in the declaration shows that Maybe can apply to any concrete type. Type constructors like Maybe differ from data constructors like Nothing or Just.

Chaining Functions and Currying

The :: notation explicitly tells the compiler what type we expect a value to have. Even though the compiler can usually infer types, it's good practice to specify types for top-level definitions. The arrow in a type signature denotes a function; a function toStrict that takes an L.ByteString and returns a B.ByteString would be typed accordingly.

> toStrict :: L.ByteString -> B.ByteString
> toStrict = B.concat . L.toChunks

A key aspect of Haskell is that functions can take only one argument. A "two-argument" function actually takes one argument and returns another function that takes the second argument. This reformulation is called currying. It allows partial application easily. For example, a function type like a -> b -> b is right-associative, meaning a -> (b -> b).

Grouping the other way — (a -> b) -> b — describes a function that takes another function as its sole argument and returns a value. Passing functions to functions is common and a strength of a language where functions are ordinary values, just like integers and strings.

Functions in Practice

Haskell functions can be defined in several ways. The function composition operator, ., chains functions together, but definitions can be more straightforward. A function with a single argument:

square x = x * x

For two arguments, you'd write:

plus x y = x + y

Currying allows partial application — feeding in the 5 to a plus function returns a new function that adds five to its argument. For instance:

addFive = plus 5
addFive 2 == 7

Alternatively, a lambda abstraction (an anonymous function) is defined using a backslash and arrow:

plus = x y -> x + y

All functions in Haskell are pure: they can't cause side effects like changing a global variable or performing I/O. This purity flows through the whole program, shaping how the Dropbox API calls are structured.

Operator Notation and Function Composition

In Haskell, operators and functions are two syntaxes for the same concept. You can use any operator in prefix form by wrapping it in parentheses — for example, addition can be written as (+) 4 5. Conversely, any function can be used in infix form by surrounding its name with backticks:

div 1 2 == 1 `div` 2

This works well in expressions like "Dropbox" `isInfixOf` "The Dropbox API is sick!". The function composition operator . takes two functions and returns a new function that applies the right function first, then the left function to that result. You can define it in infix or prefix notation:

(.) f g = x -> f (g x)

Haskell also provides sections — partial application of operators in infix form. For example, (+5) is a function that adds five to its argument. All of these are equivalent functions:

  • (+5)
  • (5+)
  • x -> x + 5
  • (+) 5
  • plus 5
  • addFive

Note that the side you curry the argument on matters — (.g) and (g.) behave differently.

The Function Application Operator

Another operator you will see frequently is $, the function application operator:

f $ x = f x

Normal function application has the highest precedence of any operator and is left-associative:

f g h j x == (((f g) h) j) x

Conversely, $ has the lowest precedence and is right-associative:

f $ g $ h $ j $ x == f (g (h (j x)))

This makes it useful as an alternative to parentheses for improving code readability.

Lists and Their Operations

> listToPair :: [a] -> (a, a)
> listToPair [x, y] = (x, y)
> listToPair _ = error "called listToPair on list that isn't two elements"

A list is a higher-order type representing an ordered collection of same-typed values. Types are denoted with brackets, e.g. [a] for a polymorphic list or [Int] for a list of integers. Unlike arrays in other languages, you cannot index into a list in constant time — it behaves more like a linked list. You can construct lists in several ways:

  • The empty list: []
  • List literal: [1, 2, 3]
  • Adding to the front: 1 : [2]

The : cons operator creates a new list starting with the left operand and continuing with the right:

1 : [2, 3, 4] == [1, 2, 3, 4]

A list is just a recursive algebraic datatype — there is nothing special about it:

data MyList a = Nil | Cons a (MyList a)

-- using "#" as my personal ":" operator
x # xs = Cons x xs

myHead :: MyList a -> a
myHead (Cons x xs) = x
myHead Nil = error "empty list"

Two essential functions for list manipulation are foldr and map. foldr means "fold from the right"; it takes an aggregating function, a starting value, and a list, visiting each element to build an aggregate. It can be defined recursively:

foldr :: (a -> b -> b) -> b -> [a] -> b
foldr f z []     = z
foldr f z (x:xs) = f x $ foldr f z xs

This deconstructs the input list into head and tail components using :. For instance, summing a list:

foldr (+) 0 [1..5] == 15

Here [1..5] is syntactic sugar for all integers 1 through 5. Less usefully, foldr can copy a list:

foldr (:) [] ["foo", "bar", "baz"] == ["foo", "bar", "baz"]

Its counterpart foldl folds from the left:

foldl :: (b -> a -> b) -> b -> [a] -> b
foldl f z []     = z
foldl f z (x:xs) = foldl f (f z x) xs

Note the reversed type signature of the aggregating function between the two. The direction of the fold matters: try copying a list with foldl as an exercise.

The map function applies a user-supplied function to every value in a list, returning a new list. It can itself be defined using foldr:

map :: (a -> b) -> [a] -> [b]
map f l = foldr ((:) . f) [] l

Even though foldr is more primitive, map is used more often in practice. For example, mapping a list of strings to their lengths:

map length ["a", "list", "of", "strings"] == [1, 4, 2, 7]

More interestingly, you can use map with curried sections to apply the same argument to a list of functions:

map ($5) [square, (+5), div 100] == [25, 10, 20]

Here, currying 5 into the right side of $ creates a function that takes a function and applies it to 5; map then applies that to every function in the list.

Tuples

Tuples also group values, but unlike lists, a tuple can hold values of different types:

intAndString :: (Int, String)
intAndString = (07734, "world")

A subtle difference is that tuples of different lengths are different types. Getting the first element of a three-element tuple is straightforward:

fst3 :: (a, b, c) -> a
fst3 (x, _, _) = x

But there is no general "first" function for tuples of arbitrary length without writing separate functions per type. The standard fst works only on two-element tuples. The underscore _ in the pattern above is a common way to ignore unused values in destructuring.

Template Haskell and Type Classes

> $(mkYesod "ImageSearch" [parseRoutes|
>                         / HomeR GET
>                         /dbredirect DbRedirectR GET
>                         |])

Yesod relies on Template Haskell, a compile-time metaprogramming mechanism similar to Lisp macros. Here, mkYesod generates all required boilerplate to connect HTTP routes to handler functions. The app defines two routes:

  1. / -> HomeR
  2. /dbredirect -> DbRedirectR

Type classes define collections of functions that work across types, comparable to interfaces in C# or Java but supporting ad-hoc polymorphism. Instances supply the actual implementations for specific types. The word "instance" in this context is different from object-oriented programming — there it is closer to a value. Many core functions belong to standard type classes, such as (==) in Eq. For example, a custom Binary type class could define conversions between a type and byte format:

import qualified Data.ByteString as B

class Binary a where
  toBinary :: a -> B.ByteString
  fromBinary :: B.ByteString -> a

An instance for String illustrates composition:

import Data.Text as T
import Data.Text.Encoding as E
import Data.ByteString as B

instance Binary String where
  toBinary = E.encodeUtf8 . T.pack
  fromBinary = T.unpack . E.decodeUtf8

Here toBinary composes E.encodeUtf8 and T.pack. The T.pack function converts a String to Text, then E.encodeUtf8 converts that to a ByteString. fromBinary does the reverse. Another instance can convert ByteString to Int32:

import Control.Arrow ((***))
import Data.Int (Int32)
import Data.Bits (shiftR, shiftL, (.&.))

-- stores in network order, big endian
instance Binary Int32 where
  toBinary x = B.pack $ map (fromIntegral . (0xff.&.) . shiftR x . (*8))
               $ reverse [0..3]
  fromBinary x = sum $ map (uncurry shiftL . (fromIntegral *** (*8)))
                 $ zip (B.unpack x) (reverse [0..3])

A critical advantage over interfaces is that type classes easily extend existing types — neither the String nor Int32 authors needed to anticipate the Binary class. Functions can also constrain their types to members of a class:

packWithLengthHeader :: Binary a => a -> B.ByteString
packWithLengthHeader x = B.append (toBinary $ B.length bin) bin
  where bin = toBinary x

The context Binary a => requires the type a to implement Binary. Note that this function also implicitly requires Int to have a Binary instance.

> instance Yesod ImageSearch where
>   approot _ = T.pack "http://localhost:3000"

Yesod requires instances for your application type. Most definitions in the Yesod type class have sensible defaults, but you must define approot — the root URL — so Yesod can generate URLs.

> instance RenderMessage ImageSearch FormMessage where
>   renderMessage _ _ = defaultFormMessage

Another instance declaration shows a multi-parameter type class RenderMessage over ImageSearch and FormMessage, defining renderMessage to return defaultFormMessage.

Handling Failures with Exceptions

Haskell offers several ways to signal errors. In pure code, Maybe and Either are common choices. But when working in the IO monad, exceptions are often more practical. To throw exceptions from the Dropbox SDK functions—which return Either—we define a custom exception type.

Like any algebraic datatype, it supports automatic type class derivation. Here we derive Show and Typeable instances, with the latter enabled by the DeriveDataTypeable LANGUAGE pragma. The final line makes EitherException an instance of C.Exception. That instance declaration doesn’t define any functions because C.Exception supplies default implementations for any type that is also Typeable.

> data EitherException = EitherException String
>     deriving (Show, Typeable)
> instance C.Exception EitherException

Monads and the IO Monad

Monads are types in the Monad type class. The class specifies two operations:

class Monad m where
  (>>=) :: m a -> (a -> m b) -> m b
  return :: a -> m a

The >>= operator—called bind—takes a monad containing a value of type a, a function from a to a monad of type b, and returns a monad of type b. The return function wraps a plain value in a monad. Crucially, return does not short-circuit or exit a block of code like a return statement in imperative languages.

Monads are a general computational framework. Their importance in Haskell comes largely from their ability to model IO in a purely functional way. A pure function must consistently map the same inputs to the same outputs. How, then, would you define a function like getChar, which takes no arguments yet must return different characters depending on user input?

getChar :: Char
getChar = ...

You can’t—not purely. Instead, Haskell generates a set of actions to perform as IO happens. This is the IO monad: a value in it describes a computation that, when executed by the runtime, performs side effects. It’s a form of metaprogramming. Consider an IO action that prints “yes” if the user types “y” and “no” otherwise:

-- type signatures
print :: Show a => a -> IO ()
getChar :: IO Char

main :: IO ()
main = getChar >>= (x -> print $ if x == 'y' then "yes" else "no")

Notice that the result of getChar isn’t defined independently—the bind operator extracts its output value and passes it onward. This is how monads encode sequential, stateful computation while remaining purely functional.

For readability, Haskell provides do notation, which desugars nested binds into a linear sequence. Here’s the same IO action with do notation:

main = do
  x <- getChar
  print $ if x == 'y'
          then "yes"
          else "no"

In do notation, the <- arrow pulls a value out of a monad, binding it to a name for the rest of the block. Monads aren’t limited to IO; their power applies broadly. The machinery itself isn’t magic—here’s a custom implementation of the IO monad:

data MyIO a = PrimIO a | forall b. CompIO (MyIO b) (b -> (MyIO a))

myGetChar :: MyIO Char
myGetChar = PrimIO 'a'

myPrint :: Show a => a -> MyIO ()
myPrint s = PrimIO ()

instance Monad MyIO where
  m >>= f = CompIO m f
  return x = PrimIO x

runMyIO :: MyIO m -> m
runMyIO (PrimIO x) = x
runMyIO (CompIO m f) = runMyIO (f (runMyIO m))

In the real IO monad, of course, getChar doesn’t hard-code its result, and print actually outputs text. IO actions are run by the Haskell runtime, which ultimately calls underlying functions written in a language like C that can perform non-pure operations.

The Either Type and exceptOnFailure

Back to the utility function that converts Either failures into exceptions:

exceptOnFailure :: MonadBase IO m => m (Either String v) -> m v
exceptOnFailure = (>>=either (C.throwIO . EitherException) return)

The Either datatype is defined as:

data Either a b = Left a | Right b

either :: (a -> c) -> (b -> c) -> Either a b -> c
either f _ (Left x) = f x
either _ g (Right x) = g x

By convention, the Left constructor holds an error value. The either function extracts a result from both cases. In exceptOnFailure, we pass a function to bind that throws an exception via C.throwIO when Left is present, or rewraps the success value with return otherwise.

Beyond that, we define helper functions to avoid repeating DB.withManager, and we use exceptOnFailure so the SDK’s IO (Either ...) results automatically throw on failure.

> myAuthStart :: DB.Config -> Maybe DB.URL -> IO (DB.RequestToken, DB.URL)
> myAuthStart config murl = exceptOnFailure $ DB.withManager $ mgr ->
>   DB.authStart mgr config murl
> 
> myAuthFinish :: DB.Config -> DB.RequestToken -> IO (DB.AccessToken, String)
> myAuthFinish config rt = exceptOnFailure $ DB.withManager $ mgr ->
>   DB.authFinish mgr config rt
> 
> myMetadata :: DB.Session -> DB.Path -> Maybe Integer
>               -> IO (DB.Meta, Maybe DB.FolderContents)
> myMetadata s p m = exceptOnFailure $ DB.withManager $ mgr ->
>   DB.getMetadataWithChildren mgr s p m
> tryAll :: MonadBaseControl IO m => m a -> m (Either C.SomeException a)
> tryAll = C.try

For broad exception handling, C.try uses ad-hoc polymorphism on the Exception type class. A explicit signature binding tryAll to C.SomeException catches any exception. One additional constant stores the session key name for the OAuth request token, which becomes relevant shortly.

> dbRequestTokenSessionKey :: T.Text
> dbRequestTokenSessionKey = T.pack "db_request_token"

How Dropbox Authentication Works

The Dropbox API uses OAuth to grant apps access to user accounts. Every endpoint call must include an access token: a revocable, per-user, per-app token issued at the user’s request. Acquiring one is a three-step process:

  1. The app obtains an unauthenticated request token from the Dropbox API.
  2. The app redirects the user to Dropbox’s website, where the user can approve or deny the request.
  3. Dropbox redirects the user back to the app, which then exchanges the authenticated request token for an access token.

A request token actually has two parts: a key and a secret. Only the key is meant for plaintext exposure. The secret must remain known only to Dropbox’s servers and the app—exchanging an authenticated request token requires presenting the original secret, which prevents third parties from hijacking the token. Access tokens are long-lived but can be invalidated at any time. If one becomes invalid, the app must run the authentication flow again.

Initiating Auth from the Web App

The web interface is implemented with Yesod, where routes are handled by functions in the Handler monad. getHomeR serves the root path / for GET requests. The Handler monad behaves like a decorated IO monad, so handlers can use standard IO-style code with liftIO to pull IO actions up into Handler.

> getHomeR :: Handler RepHtml
> getHomeR = do

Inside the handler, getYesod retrieves the app’s value, which has the ImageSearch type defined earlier. From it we extract just the config, which holds the app key and locale the Dropbox SDK needs.

>   ImageSearch _ dbConfig <- getYesod

Next, getUrlRender supplies a function that turns route values into plain textual URLs. We call myAuthStart—a wrapper around DB.authStart—passing DbRedirectR, the URL where the user should land after authorizing. The function returns a fresh, unauthenticated request token and the Dropbox URL for the user to authorize it. Note that T.unpack converts Text to String, while T.pack does the reverse.

>   (requestToken, authUrl) <- liftIO
>                              $ myAuthStart dbConfig
>                              $ Just $ T.unpack $ myRender DbRedirectR

From the token we pull out the key and secret, storing both in the Yesod session—an encrypted set of cookie-based key-value pairs—with setSession. We’ll need the secret later to complete the exchange.

>   setSession dbRequestTokenSessionKey $ T.pack $ ky ++ "|" ++ scrt

Finally, we redirect the user to authUrl. After the user makes their choice at Dropbox, the request token is authenticated (or denied), and the user is sent back to the redirect URL, where we’ll be ready to finish the OAuth dance.

Turning the OAuth Callback into a Token

After Dropbox redirects the user back to our site, the callback handler must extract the query arguments (oauth_token and uid) from the GET request. Yesod provides runInputGet for exactly this purpose.

The pair constructor (,) is a special prefix-only operator. To combine two parsed values into a pair within the applicative context, we use the <$> and <*> operators:

(,) x y = (x, y)

These operators belong to applicative functors — a weaker cousin of monads. A useful mental model is to define them in monad terms:

(<*>) :: Monad m => m (a -> b) -> m a -> m b
mf <*> m = do
  f <- mf
  x <- m
  return $ f x

(<$>) :: Monad m => (a -> b) -> m a -> m b
f <$> m = return f <*> m

If you remove >>= from the picture and rely only on <*> and pure (the applicative version of return), you have an applicative functor rather than a monad:

class Applicative f where
  pure  :: a -> f a
  (<*>) :: f (a -> b) -> f a -> f b

The actual Applicative type class definition is slightly more involved, but this simplification captures the point. In getDropboxQueryArgs, we apply the pair constructor in the applicative functor and pass the resulting value to runInputGet, which executes it in the Handler monad.

This pattern can take some time to internalize, but the behavior is straightforward: build up a computation that extracts multiple values, then combine them.

Verifying the Callback

The authentication result handler lives at the redirect destination. Before sending the user to Dropbox, we stored a request token in the Yesod session with setSession. After the redirect, we retrieve it with lookupSession, which returns a Maybe value:

>   let noCookieResponse = defaultLayout [whamlet|
> cookies seem to be disable for your browser! to use this
> app you have to enable cookies.|]
> 
>   when (isNothing mtoken) $ noCookieResponse >>= sendResponse

The when function from Control.Monad runs a monadic action only when its first argument is True. If mtoken is Nothing — meaning no session token exists, likely due to disabled cookies — we return an error via sendResponse and halt normal processing.

Once we confirm the token exists, we extract it with fromJust and split it on the "|" delimiter using T.splitOn. The @ pattern syntax lets us bind the full split list to rt while also naming the first element sessionTokenKey:

>   (getTokenKey, _) <- getDropboxQueryArgs
>   when (getTokenKey /= sessionTokenKey)
>     $ invalidArgs [T.pack "oauth_token"]

We then obtain the request token key from the query arguments via getDropboxQueryArgs. Comparing this against the stored session key protects against cross-site request forgery. A mismatch triggers invalidArgs, stopping the handler. This ensures only Dropbox — not a malicious third-party site — can invoke this endpoint.

Next, listToPair converts the two-element list into a tuple. The uncurry function unpacks that tuple into two separate arguments for the DB.RequestToken constructor, with map T.unpack converting the Text values to String:

>   ImageSearch chan dbConfig <- getYesod
>   accessToken <- liftIO $ myAuthFinish dbConfig requestToken

With the authenticated request token in hand, we call DB.authFinish (via myAuthFinish) to obtain the access token. This app uses Concurrent Haskell, so multiple threads of control run within the IO monad. The two main threads are the web server (handling HTTP requests) and the background process that updates user Dropboxes with search results. Haskell channels provide typed inter-thread communication; we send the access token over a channel so the updater thread can begin processing for this user:

>   defaultLayout [whamlet|okay you are hooked up!|]

Arrows in Practice

Beyond monads and applicative functors, arrows offer another composition pattern. They model operations that convert input to output — conceptually similar to filters. Arrow types implement the Arrow type class:

class Arrow a where
  arr :: (b -> c) -> a b c
  (>>>) :: a b c -> a c d -> a b d
  first :: a b c -> a (b, d) (c, d)

The actual type class is more complex, but this outline is sufficient. Regular functions are the canonical example of arrows — in fact, functions are instances of the Arrow type class:

instance Arrow (->) where
  arr f = f
  f >>> g = g . f
  first f (x, y) = (f x, y)

The arrow symbol -> is an operator in Haskell's type language, with a prefix form like any other operator: (->) a a is identical to a -> a. Declaring instance Arrow (->) turns the infix type operator into its prefix form to define the instance for functions. In the type language, -> is a type constructor that creates a function type when applied to two types.

Combinator libraries frequently leverage the Arrow type class to compose domain-specific functions intuitively. HXT is one such library for XML processing. In our app, we define the selectImageUrls arrow, which takes an XmlTree document and extracts all links containing the string imgurl= — these are the embedded URLs in the image search result page pointing to the actual image files.

Thread Management

The image search functionality relies on many threads, not just one. A dedicated thread listens on the channel for users linking their accounts. Each user gets a thread polling their Dropbox every 30 seconds for new folders to populate. For each new folder, another thread handles the upload of search results.

In languages like C, C++, Java, or Python, spawning unbounded threads is inefficient because they map closely to kernel-level threads, which don't scale well into the tens of thousands. Modern GHC, by contrast, keeps Haskell threads cheap and distributes them across a small pool of kernel threads — typically one per CPU — making thousands of concurrent threads entirely feasible.

The Folder Upload Worker

handleFolder is the thread responsible for populating a specific folder in a user's Dropbox with the image results:

>   let searchTerm = takeFileName filePath

The file name portion of the folder path serves as the search term. The variable src holds the fully constructed image search URL, generated with the URLEncoded library to properly escape the query string.

Fetching the search result page uses simpleHttp, which returns a lazy ByteString. Since the XML library requires a String, we convert with a combination of T.unpack, decodeUtf8With, and toStrict:

>   images <- runX ( readString [ withParseHTML yes
>                               , withWarnings no
>                               ] imageSearchResponse
>                    >>>
>                    selectImageUrls
>                  )

The HXT library parses out the relevant image URLs via the selectImageUrls arrow, producing a list of type [String]. We then iterate with forM_, performing an IO action for each URL. Wrapping each step in tryAll prevents an exception in any single image from aborting the entire upload process:

>     urlenc <- URL.importURI $ fromJust $ URI.parseURIReference url
>     let imgUrl = fromJust $ URL.lookup ("imgurl" :: String) urlenc
>         imgUrlURI = fromJust $ URI.parseURIReference imgUrl
>         imgName = takeFileName $ URI.uriPath imgUrlURI
>         dropboxImgPath = combine filePath imgName

Each parsed URL contains the source image URL within an embedded query argument, imgurl. From this we extract imgUrl and construct dropboxImgPath, the destination path in the user's Dropbox. simpleHttp fetches the source image body as a lazy ByteString:

>     DB.withManager $ mgr ->
>       DB.addFile mgr session dropboxImgPath
>       $ DB.bsRequestBody $ toStrict image

The Dropbox SDK call DB.addFile uploads the file data to the specified path. If a file already exists there, the method will not overwrite it.

Watching for New Folders

Each linked user gets a handleUser thread responsible for monitoring their Dropbox for new folders that should be populated with image search results. The thread polls the Dropbox every 30 seconds and loops indefinitely.

> handleUser :: Chan DB.AccessToken
>               -> DB.Config
>               -> DB.AccessToken -> [String] -> IO ()
> handleUser chan dbConfig accessToken_ foldersExplored = do

It’s possible for the handleNewUsers thread to push a new access token to a running handleUser thread through the channel passed in via the chan argument.

>   accessToken <- let getCurrentAccessToken x = do
>                        emp <- isEmptyChan chan
>                        if emp
>                          then return x
>                          else readChan chan >>= getCurrentAccessToken
>                  in getCurrentAccessToken accessToken_

The inner function getCurrentAccessToken, defined with let ... in ... syntax, drains the channel using isEmptyChan and returns the last token that was pulled off before the channel went empty.

>   efolders <- tryAll $ do

IO actions in this loop are wrapped to catch transient exceptions.

>     let session = DB.Session dbConfig accessToken

The DB.Session value, bound to the name session, is what the Dropbox SDK needs for uploading file data into a user’s account.

>     metadata <- myMetadata session "/" Nothing

The code uses myMetadata to retrieve the collection of children inside the root of the API app sandbox (/).

>     let (_, Just (DB.FolderContents _ children)) = metadata
>         folders = mapMaybe ((DB.Meta base extra) ->
>                              case extra of
>                                DB.Folder -> Just $ DB.metaPath base
>                                _ -> Nothing) children
>         newFolders = folders DL. foldersExplored

The mapMaybe function combines map and filter: elements for which the input function returns Nothing are dropped, while Just values are unwrapped and kept. Here it extracts the subfolders in /, excluding plain files. The DL. operator performs a set-difference operation, returning all elements in the first list that don’t appear in the second.

>     forM_ newFolders (forkIO . handleFolder session)

For each folder that is new, the code again uses forM_, this time spawning a dedicated thread via forkIO.

>     return folders

The updated path list is returned to the parent IO action so it can track which folders have already been processed.

>   threadDelay $ 30 * 1000 * 1000

threadDelay pauses execution for 30 seconds, much like sleep() in other languages.

>   let curFolders = either (const []) id efolders

If polling fails, an empty list is bound to curFolders; otherwise the current folder listing is used.

>   handleUser chan dbConfig accessToken
>     $ curFolders `DL.union` foldersExplored

After that pause, the function recurses to loop again. Recursion is how loops are expressed in a monad. Before recursing, however, the total set of seen folders is updated so the next iteration won’t attempt to populate them again.

Handling New App Users

The handleNewUsers thread listens on a channel for accounts that have just linked to the app and starts a handleUser thread for each one.

> handleNewUsers :: ImageSearch
>                   -> Map.Map String (Chan DB.AccessToken)
>                   -> IO ()
> handleNewUsers app_@(ImageSearch chan dbConfig) map_ = do

Pattern matching with the @ syntax binds the whole ImageSearch value to app_ while also deconstructing it into its chan and dbConfig components. The map_ argument is a mapping from user ID to the channel of the thread responsible for that account, so a revoked token can be refreshed in-place.

>   (accessToken, userId) <- readChan chan

readChan pulls a tuple — user ID and access token — off the channel shared with the web server thread.

>   newMap <- case Map.lookup userId map_ of
>     Just atChan -> do
>       writeChan atChan accessToken
>       return map_
>     Nothing     -> do
>       nChan <- newChan
>       _ <- forkIO $ handleUser nChan dbConfig accessToken []
>       return $ Map.insert userId nChan map_

The user ID is looked up in the thread-channel map. If a thread already exists for this account, the new token is sent on its channel. Otherwise, a fresh channel is created, a handleUser thread is forked, and the map is extended with the new entry.

>   handleNewUsers app_ newMap

Finally, the function recursively loops with the updated map.

Program Entry Point

> main :: IO ()
> main = do

The main IO action is where execution begins for any Haskell program, analogous to main in C/C++ or Java.

>   let defaultAppKey = undefined :: String
>       defaultAppSecret = undefined :: String
>       defaultAppType = undefined :: DB.AccessType

Default app credentials are declared as the polymorphic constant undefined (often written _|_), which can inhabit any type but throws an exception the moment it’s evaluated. For this demo they are inlined rather than parsed from the command line or a config file, and you must substitute real values before the app will run.

>   chan <- newChan

An inter-thread communication channel is created with newChan.

>   dbConfig <- DB.mkConfig
>               DB.localeEn
>               defaultAppKey
>               defaultAppSecret
>               defaultAppType
>   let app_ = ImageSearch chan dbConfig

The channel and a DB.Config value are combined into the app-specific ImageSearch record.

>   _ <- forkIO $ handleNewUsers app_ Map.empty

The handleNewUsers thread is started to accept new account links.

>   warpDebug 3000 app_

warpDebug then launches the Yesod web interface.

Assessment

The app is far from production-ready. It has no way to persist which users have linked or which folders have been populated across restarts, so that state would need a proper database.

The 30-second polling interval creates a poor worst-case user experience. More importantly, it causes the load on the Dropbox API to scale linearly with the total number of linked users rather than with the number of active ones. While this is an acknowledged problem with ongoing work, other refinements such as a better folder-selection algorithm, improved HTML for the web interface, and streaming uploads to the API are also mentioned as natural next steps.