Token Bucket Algorithm

This was written via Markdown. Apologies if its unclear, or your glasses are missing 😅.

Given:

  • a bucket b of a specific size n.
  • a token i representing a unit of consumption e.g. each token may represent an object/element such as # of API requests, # of TCP packets, # of order basket item quantities, integer prices etc. Basically, what you want “consumed” from the recipient.
  • a token refill rate which determines how fast rr tokens are added to the bucket per t seconds (can be larger time periods e.g. minutes, hours, days).
  • a consumption rate r which determines how fast i tokens are consumed from the bucket. This can be an API request rate per second, purchase requests per second etc.., each having a given number of “tokens” they want to consume per given time period.

The formula is (as far as I understand) as follows:

  1. the bucket b is initialized with n number of tokens. If not initialized with initial tokens, all initial requests will fail until at least 1 token is available for consumption, given requirement #3 below.
  2. For every consumption request, a given number of tokens #i is consumed from the bucket.
    • if #i is equal to or more than n, the tokens are consumed, and the remainder calculated and saved back into bucket b as n for this request. The request passes.
    • if #i is less than n, the request is ideally denied, but other mechanisms can be applied e.g. waiting for i tokens to be available. So n remains untouched for this specific request. The request fails.
  3. Given the rate which determines how fast tokens are refilled i.e. if every rr/t period of time passes, a single token is added to the bucket, up to a maximum of size n:
    • if the rate of refill is 5 tokens every 10 seconds, divide 5 tokens / 10 seconds, and you get 1 token every 2 seconds on best effort (i.e. depends on availability of resources for every action/request). So if your consumption rate is 10 tokens every 10 seconds, that makes your consumption rate r to be 10 tokens/10 seconds, thus r == 1/s. Bucket can only refill at effectively 5 tokens/10 seconds, making the rr/t == 0.5/s i.e. 1 token every 2 seconds, thus half your requests will probably fail.
    • if the rate of refill is 20 tokens every 5 seconds, divide 20 tokens / 5 seconds, and you get 4 tokens every 1 second on best effort. Thus if your consumption rate is 10 tokens every 10 seconds, that makes your consumption rate r to be effectively 10 tokens/10 seconds, thus r == 1/s. Bucket can be refilled at 20 tokens every 5 seconds, that makes the effective minimum 4 requests/1 second i.e. rr/t == 4/s. All your requests should probably go through.

Disclaimer

  • This has NOT taken into account bursts! The above is an ideal situation which will happen as part of daily life, but definitely expect the bursts! I might add that section later.
  • Race conditions are also possible, so beware!

Why I like this algorithm? Don’t be greedy 😁. BUT! When you have an audience whose usage you can determine or predict beforehand, this algorithm makes great sense to me! Payments APIs, Rate limiting for uploads, static content which might be exposed, any requests for sensitive info etc..

The Human Limits

Light is the fastest thing in the universe so far as humans are aware, and it’s natural (how we describe whatever humans discovered existed, have no idea how it came to be, or works sometimes (my opinion)).

Yet we humans have made faster things than nature:

  • Land:: cheetah: car?
  • Air:: falcons: planes?
  • Water:: black marlin: not sure, but submarine and speedboat are close (?)🤷🏽‍♂️
  • Space: up for grabs!!

Correct me where appropriate

Light is natural, humans will find something faster, with time…

We just have to survive ourselves…

M-Pesa / Daraja API Credentials

This is one of those articles you write when you have just had it, when like a hundred people keep asking the same question over and over again and can’t seem to understand, or just don’t want to do the work, and want to be spoon fed, or simply just can’t understand what the heck to do about it or go about it. I hope this article clears as much of this as possible.

This is about M-Pesa’s credentials, in the Portal and outside on their Daraja API. And here, the word Credential should be taken to mean any form of code or security that seeks to authenticate the user to Daraja or M-Pesa, no matter the type or format, for the user to be able to access its services. This article is also very specific to Daraja.

M-Pesa basically has 2 sets of credentials:

  • Portal Credentials – these are used by users on the M-Pesa portal, also known as operators.
  • API credentials – these are credentials created for use in API calls to the M-Pesa API via Daraja.

There are other credentials on other older M-Pesa APIs, but they are not our focus here.

Portal Credentials

When you access the M-Pesa portal (located at https://org.ke.m-pesa.com/), you have a user whom you use to log in. That user has a password which you either assigned to them if you created them, or was given by Safaricom. That user is also known as an operator, and more specifically a web operator. That means they are allowed to access M-Pesa via the web portal. Their password represents the first type of security credential for M-Pesa, a simple password backed by a one time verification code for every login. This password is as simple as they come. The only requirement is that it has at least one special character, one capital letter and one digit. Note that M-Pesa does not recognize the ‘@’ character as a special character.

For a web operator, their password has an expiry period, around 3 months (last I checked). That means you gotta keep your password updated regularly, or you will be locked out and the web operator’s status changed to dormant. When that happens, if you are not the Business Administrator (this is a role assigned to the first primary operator of a shortcode), you can have the Business Admin for your shortcode re-enable you, otherwise you gotta get in touch with Safaricom M-Pesa Business team to re-enable you. If you plan to use a web operator for API requests, always remember to renew the password before the expiry, otherwise all requests will fail with the “The Initiator Information Is Invalid” error.

Now, this tutorial assumes you want to integrate into the M-Pesa APIs. And that means you will need an API operator. This is the API equivalent of the web operator, meaning they are only allowed to perform API calls, and can never log into the web portal. Their passwords also never expire (last I checked). They are created by the Business Admin of a shortcode to enable the business owner to perform API transactions. Once created, they are then also known by their API name: Initiators. Thus their assigned usernames become known as Initiator Names and the assigned passwords become Initiator Passwords. Remember you have to assign them the right roles to be able to perform the required API calls.

For the API operators/Initiators, their passwords represent the second level of credentials for M-Pesa. The passwords need to be encrypted using an RSA Public Key certificate provided by Safaricom before being used in an API. The Certificates are available in PEM on the developer portal documentation. Once encrypted, the resulting cipher-text is then what is used as the SecurityCredential or Initiator Encrypted Password in the body of API requests to Daraja and other M-Pesa APIs e.g. in the Daraja API B2C Sample request below, the SecurityCredential parameter contains the cipher-text resulting from the encryption process described mentioned above:

{
    "InitiatorName": "apitest361",
    "SecurityCredential": "iDIurkCG4r9F7lCCNNkctseY9MgiddL6F1VmmJH1DSk5n5U4HPkE3gj1+UFNQ+3MUIzIkR7EgBov9/rqzeXJVb8JeTTD/10XSdTU5eJJeWsqI1VSXpIe/j4xa7HNuezxRJ/KtTWu3bdq9heozyXMn/5ErogjfwIlfQ4Bkhw8+9SEGoNkRA95Idp1PSg/ElPOVnrKuPFTf0HS/iTmuKvVBkY6oAFSObVx7DmqqoL2K4P8ZNh3EDVfQtCQGUnF1OLOab5X46wqVW5p7UJ/5kVFdN8Yuw172VNOjyNILjUXwqqAhS4WKXC28cXd4faYrG5E5AQfGGjNIlYRqWse8w23fw==",
    "CommandID": "[CommandID]",
    .... //other data
}

That is the ONLY place you ever use an encrypted password in API requests to M-Pesa. Nowhere else! Remember that.

Those are the two sets of portal credentials you can expect to work with on the M-Pesa portal side. Moving to the APIs…

API Credentials

When you get to Daraja, you are expected to authenticate your API calls to Daraja (if you don’t know what am talking about please take a detour). Thus authentication on Daraja takes us to the other set of credentials: API credentials.

Note that the API operator/initiator credentials are created on the portal, but used on the APIs only.

When sending any request to Daraja, you need to send a specific header as part of the headers. This is the Authentication header. For almost all API calls to Daraja, you need to first fetch an access token that will be authenticated against the API. To get this token you use the Generate Token API. This API requires you to send the same Authentication header parameter to get the access token but with a different value. That header is an M-Pesa API credential, and is made up of base64 encoded consumer key and consumer secret from an App of your choosing on the developer portal, joined together by a full colon (again, all that is here). That header must always be present in every request, under the Authentication key, and expires after 3600 seconds, or 1 hour. So keep in mind to refresh it regularly. Note that you only encode the consumer credentials, and that you only ever use the tokens in the HTTP header, nowhere else. The screenshot below shows where the header is applied when making an API request from the portal. The <Access-Token> variable is replaced with the result of the base64 encoding described above. Remember to always include the prefix required in the header e.g. the word Bearer in the screenshot, followed by a space, then the token. Otherwise requests will fail with an “Invalid Access Token” error

When used in actual API requests, the header is usually used in the format Authorization: Bearer <Access-Token>. So take note of how to include headers in your selected language / environment.

Last, but definitely not least, we have the Lipa na M-Pesa password. This is a credential that is only used in the body of Lipa na M-Pesa requests. To use or apply this credential, you first need to get a passkey from Safaricom, or if on the sandbox, use the provided passkey (this seems to no longer be the case for the new Daraja portal). Then create the password by base64-encoding your concatenated shortcode + passkey + timestamp. That gives you the value to use in the Password parameter of LnM requests. Sample usage is shown below:

{
	"BusinessShortCode": "174379",
	"Password": "MTc0Mzc5YmZiMjc5ZjlhYTliZGJjZjE1OGU5N2RkNzFhNDY3Y2QyZTBjODkzMDU5YjEwZjc4ZTZiNzJhZGExZWQyYzkxOTIwMjAwNDMwMTgzOTQ5",
	"Timestamp": "20200430183949",
        .... //other data
}

Note that the Timestamp value in the request must be the same as the timestamp used in the creation of the Password. Also the timestamp must be in the format YYYYMMDDHHmmss. Otherwise you also get the “MerchantValidate – Wrong credentials” error. The Date format pattern usually differs between languages, thus the base rules are:

  • The year must be 4 digits long
  • The Month should use its numeric value, not name e.g. January is 01, not Jan, January or whatever else you are thinking about.
  • The Day is the day of the month, between 1 and 31.
  • The Month, Day, Hour, Minutes and Seconds must be 2 digits long, prefixed by 0 (zero-padding) where needed.
  • The hour should be 24-hour format.

And that’s the only place you ever use the LnM password.

Thus, in summary, you pretty much always use either one or two credentials when working with M-Pesa APIs. If you ever find yourself using all three, just delete all your code and start over.

If I can summarize the APIs and their usage of the credentials, it would be as below:

APIM-Pesa Initiator PasswordAuthorization Header (Type)LnM Password
Generate Token ✔ (Basic)
Reversal ✔ (Bearer)
Transaction Status ✔ (Bearer)
Account Balance ✔ (Bearer)
B2C ✔ (Bearer)
C2B (Register URL) ✔ (Bearer)
M-Pesa Express (LnM) ✔ (Bearer)

I have tried and created sample codes on how to generate each type of credential for Java, PHP and JS. Unfortunately for JS I haven’t found a way to encrypt the password from the server side, thus the available code only works on Browser side. If you have sample Node code to encrypt the M-Pesa Initiator password, please host on GitHub and share a link, I shall put it here for others to check out:

Again, if none of the above make sense, start here.

Note that the certificate in the repos above is the G2 API certificate, thus will most probably not work with Daraja. Download a fresh copy of the certificates from the Daraja portal to work with on Daraja.

That should give you a good head start in that stuff. Also note the very generous use of the word only for those peeps who use a single credential everywhere they want. Now, I don’t wanna hear anymore credentials questions.


For all M-Pesa APIs consultation, contact Peter via consult@peternjeru.co.ke

Pay via ProxyAPI on WooCommerce

Hey All! We now have a new Plugin available for WooCommerce Clients and Admins backed by Proxy API!

The new plugin enables one to accept Lipa na M-Pesa payments on their WooCommerce sites powered by Woo Pay via ProxyAPI plugin. The plugin is integrated to use the PVP functionality introduced in the PVP API and this enables all payments made via the plugin to be initiated and completed automatically. That means the plugin also handles callbacks from Proxy API and updates the orders with the current transaction details at the back of it all.

Woo Pay via ProxyAPI (WPVP) Plugin does not keep a separate table for MPesa transactions. It receives and updates transaction results directly on the Order, allowing for a more convenient way to keep track of orders. All MPesa transactions are directly tied to their orders via the Transaction ID, which allows an Admin to easily see as soon as they open the order on their administration portals. A sample output is shown below:

Note the M-Pesa Transaction ID directly tied to the order on completion after the Payment Type description

WPVP also enables the transaction to be directly tied to its CheckoutRequest ID as received from LnM API, and one can use this to query the status transaction later if the transaction is not automatically updated within a few moments after a user responds to the Safaricom payment prompt. The callback also attaches any extra data received from Safaricom to the order. All this data is easily viewable from the order metadata as shown below:

Metadata of the transaction received from MPesa.

WPVP also has custom actions which a developer can take advantage of to initiate requests after either of three events have occurred:

  • When a payment request has been sent successfully to LnM API. This event is named ‘proxyapi_pvp_payment_pending‘ and has four parameters:
    • The orderId for the current order
    • A unique RequestID created by the Plugin for the current request
    • The MerchantRequestID and…
    • The CheckoutRequestID, both received from LnM API after sending the request
  • When a payment callback has been received, and the callback indicates a failure. The event is named ‘proxyapi_pvp_payment_failed‘ and has three parameters.
    • The orderId
    • The ResultCode and…
    • The ResultDesc, both received from LnM API when a transaction has failed.
  • When a payment callback has been received, and the callback indicates a success. The event is named ‘pvp_on_payment_complete‘ has only one parameter, the orderId of the successful request.

This events will allow you to hook into the payment process and perform any other events after the order has been updated accordingly. Some sample code on how to use the above hooks is available in the Plugin Repo on Github.

FYI, an order is automatically set to failed when an error code has been received from Daraja, and automatically set to completed when a transaction ID and confirmation callbacks have been received from either Daraja or Proxy API. There is no manual action needed for either of the two and no manual checks of transactions required either unless the transaction has not been updated for more than 5 minutes, indicating LnM API was unable to send an error callback back to the plugin.

Note that the Order Number is also used as the Account Reference when the LnM request is sent to M-Pesa. The user’s phone number is also mandatory, thus beware not to remove or skip the phone number field during checkout.

On top of it all, you also get the benefit of using Proxy API portal to confirm a transaction was sent and the callback also received if in doubt.

And v2.0.0 now has a new custom M-Pesa Report tab under reports to show you the latest received transactions from ProxyAPI.

A sample WooCommerce application showcase can be found here

For details how to on-board to Proxy API, check out the documentation

For a step-by-step process on how to onboard your Paybill or Till onto ProxyAPI, check out this detailed article written by one of our clients: https://www.thebrands.co.ke/2020/09/13/integrate-m-pesa-to-woocommerce-free-plugin-and-a-step-by-step-guide/

Get it on WordPress!


For all M-Pesa APIs consultation, contact Peter via consult@peternjeru.co.ke

Pay Via Proxy API

Do you have an online business, store or website?

Do you have a Safaricom Till Number or Paybill Number?

Do you wish to integrate into M-Pesa or Lipa na M-Pesa and receive payment notifications automatically from your site?

Well, Proxy API has the right service for you!

We have developed a new integration named Pay via Proxy API that enables merchants and developers with online stores to easily integrate into Safaricom’s MPesa. The setup involves simply integrating a single button on your site that enables the client to initiate a Lipa na M-Pesa request from your site. By clicking on the button, a new window is opened, the client enters their phone number and an instant payment request is sent to their phone to complete the transaction. Simple, right?

Sample PVP LnM Request

This integration also combines Lipa na M-Pesa API and C2B API whereby the 3rd party/merchant or developer receives two callbacks: one from from Daraja and the other from Proxy API; or at least one callback from either, on successful completion of the request. This enables one to have more stability on their system and not have missing callbacks sometimes seen when using the Lipa na M-Pesa (LnM) API alone. This also means, for any unconfirmed callback, the 3rd party can check for the missing transaction on the Proxy API portal, similar to Query Checkout API on Daraja. This is a feature that will be integrated soon.

After initiating the request, the 3rd party can expect either one or both callbacks from Daraja or Proxy API. The callback from Proxy API comes from the C2B integration that will be automatically handled by Proxy API. This enables one to crosscheck both the LnM callback and Proxy API callback for the details of the transaction, whichever comes first.

Check out more info on the documentation available here.

Then begin onboarding here pap! 🙂

Safaricom Tills API Integrations

To all merchants and developers out there who own or have Safaricom Tills and who wish to integrate into Safaricom APIs, this is for you.

If you wish to integrate into MPesa API, you cannot use a Till number alone.

Tills are like small kids or even babies around money. Ideally, people can give babies and small kids any amount of cash they wish, but they do not have the facility to store it themselves. That is done by the parent of the child who has access to an account. For Safaricom Tills, that parent is a shortcode known as the Store Number. It holds the accounts where all the cash paid to a Till is deposited into. It is the shortcode which can access or be accessed by APIs. Tills have no accounts of their own, and don’t have access to any portals, thus you can’t log into the MPesa Org. Portal using Tills. Only the parent Store Number has those. Tills only act as quick facilitators of cash collection for the Store Number and also for organizational purposes.

Note also, there is a 1-to-1 mapping between the Till and the Store Number. One Till can only have one parent Store Number, and one Store Number can only be mapped to 1 Till

So make it a point, when applying for a Till Number, to also request for the Store Number and apply for an Administrator for that Store number, if you ever plan to integrate into APIs in future. That Administrator will be used in the Go Live process on the APIs and also to access the MPesa Org. Portal.

Note, there is also another shortcode above the Store Number known as the Head Office number, or H.O. for short. This is at a “grand-parent” level to the till, and can never be used for APIs, as it is a grand-collection account for the overall parent organization. Money collected by the Store Numbers via their Tills is propagated upwards during settlement processes e.t.c. One H.O. can have multiple Store Numbers under it e.g. those large supermarkets use that hierarchy to map out tills to a parent/main headquarters somewhere in Town, and whatever those guys earn, can be propagated upwards to the headquarters’ H.O. This is most probably owned by those guys who give you the Till numbers, so don’t even bother checking it out.

So, when you are using the MPesa API, whether Proxy API or Daraja, during Go Live, you should have access to and utilize the elements of the Store Number, not the Till.


For all M-Pesa APIs consultation, contact Peter via consult@peternjeru.co.ke