osCommerce

The Saferpay payment module I first worked on for the new post-transaction action features of 3.0 was implemented with a program Saferpay provides that is used to communicate to the payment gateway. The use of this program is similar to using cURL but had the disadvantage that it had to be manually installed and that the output of the program had to be saved to a file.

Creating the files was no problem as it was easily done in the osCommerce working directory, and deleted as soon as it was no longer needed.

I previously blogged about implementing an additional Saferpay module that uses SOAP to communicate to the payment gateway, which did not require the use of the “saferpay” binary.

When I started to look into it, I did not understand the documentation provided of how the system works and asked Saferpay for help. They suggested that I use a more simpler approach as setting up the module would have been too technical for store owners to achieve, and forwarded me the necessary documentation. I did not know of this simpler method beforehand and was thrilled that they provided yet another means of being able to communicate to their gateway.

The simpler approach uses normal HTTPS POST connections to communicate to their gateway, and is common for payment service providers to use.

I agree that setting up a SOAP based module would have been too technical seeing that most store owners that use the PayPal IPN module have not set it up with the Encrypted Web Payments feature it supports. And that, in contrast, is pretty easy to setup.

I’ve now updated the Saferpay Credit Card, ELV (bank transfer), and Virtual Terminal payment modules to use the HTTPS POST method, with all 3 modules retaining the same functionality and post-tranaction actions as introduced earlier.

Or maybe it is just me being a goofball… but I have just spent a bunch of money trying to make a Betta happy. Bettas are those colorful fish you see in Walmart or petstores that most people keep in a small bowl. Well, the other day my husband and I were in Walmart and we were in the pet section getting food for my dogs and I spotted the Bettas all lined up in these tiny little plastic bowls with white covers. I am such a softy and talked my husband into letting me get one. Like we need another pet. We have 4 dogs and 2 cats as it is. Anyhow, to make the fish happy I HAD to have a bigger tank so we got a 1 gallon tank, some pretty glass stones and a plant. Not bad, everything included about $40.

Ok, get home fish is in the new tank and happy. I am happy cause the fish is entertaining. Had to name him because Bettas have amazing personalitys. Like little bitty dogs or something. So anyhow, named the fish Bart. He seems really happy for a couple of days. Wouldn’t you know it though… poor thing get’s bored and keeps attacking his shadow and swimming in circles. I have had several Bettas in my life but never one that had so much energy so I had to have a bigger tank for him. HAD to. Did I mention I am a goofball?

Now, 3 days go by and I talk my husband into taking me to the pet store to get a bigger tank but I don’t feel good so he goes by himself. He comes home with a 5 gallon tank, another plastic plant, a big rock, more gravel and an algae eater Yay! Get Bart, that’s the Betta, into the new tank and he is like so psyched. He’s swimming in the stream of the filter like some little under water surfer having loads of fun. I am happy cause he is happy. 2 days go by and he is no longer swimming around all happy but spending most of his time bugging the algae eater trying to make him swim so I think he needs some friends. I really think he sees “sucker” written on my forhead.

Two days later after watching Bart act all depressed we go to the petstore and get him 4 friends. No idea what kind of fish they are but they are small and kind of cute. Well, Bart doesn’t like two of them so we have to give them to the sister in law. So now the tank has Bart, Zig and Zag… two speedy little yellow fish, and the algae eater without a name.

You would think things are perfect now but no. Things are great for about a day and a half. Because it is a new tank and plastic plants the algae eater has nothing to eat and is like acting nutso so the husband stops at a petstore on the way home from work today and picks up 2 live plants and algae eater food. omg who ever heard of such a thing? And because the fish won’t eat the food that falls to the bottom he also picked up a crab. Yes, a little crab. I named him Gary. That thing can eat and eat… he hasn’t stopped since we put him in the tank.

Well so far so good but it has only been a couple of hours. Bart is loving one of the plants the algae eater is liking the other. I have to wait until later on to put the tablet food in and see if he will eat that. Zig and Zag are well…. zigging and zagging all over the place and Gary the crab is cleaning the tank and pretty funny to watch.

So to make a Betta fish happy, this insane sucker of a woman has spent just about $150. I think I need to see someone about my softness but it is almost my birthday so I have told the husband he doesn’t have to get me anything. Like he was going to after spending all this money on a fish anyhow. They are fun to watch and sitting right here on my desk so that is a plus. I catch myself just sitting here watching them for long periods of time. Kind of relaxing and honestly more fun than most of the stuff on tv. Zig and Zag like to chase my led light I shine in the tank and Bart comes up to the surface and “kisses” my finger if I put it in the water… yes, I am easily entertained. This story should have already clued you in.

As a side note… anyone who tells you Betta fish can be happy swimming in a bowl the size of a coffee cup is just plain old wrong. They just don’t know any better.

The checkout procedure has been worked on again to meet the requirements of some payment modules, with the ordering process now being split up into numerous class methods. The logic has returned to the order class (now osC_Order) after being moved over to the checkout/process content module.

This provides a level of flexibility where it is no longer required to go through the checkout/process page to process an order, and can now be done practically anywhere.

The class methods now used to process an order currently consists of:

osC_Order::insert()
osC_Order::remove()
osC_Order::process()
osC_Order::sendEmail()

The new logic now creates the order in the database via the insert() method before contacting the payment gateway for the transaction to occur. This allows the correct order ID to be passed with the transaction request to the gateway, and only processes the order if the transaction was successful. (This replaces what I blogged about previously regarding the random and unique order transaction ID)

For payment modules that require no external interaction, such as Authorize.net, the calls to insert() -> process() -> sendEmail() are performed in the payment modules process() method.

Payment modules to date have had before_process() and after_process() class methods which has now been replaced with one single process() method.

The execution is different for payment modules that do require external interaction, such as PayPal, where the order is inserted into the database on the checkout/confirmation page, and processed at a later stage.

This behaviour was introduced with the official PayPal IPN payment module contribution made for the 2.2 Milestone 2 release, and does have some drawbacks due to technical related issues.

As most community members are aware, if a customer makes an order through the normal PayPal payment module and does not hit the last continue button on PayPal’s site to return back to the store, the order would not have been created in the database and the store owner receives a payment email from PayPal without an order to reference it to. This problem was addressed with the PayPal IPN contribution where the order is inserted on the checkout/confirmation page and processed on the callback request PayPal makes to the online shop that is performed when the payment has been confirmed by the customer.

The reason the order is created on the checkout/confirmation page and not when the customer returns to the store after paying at PayPal or when the callback request from PayPal is received, is incase the customer does not hit the last continue button on PayPals site to return back to the store, and incase the session validation features of osCommerce prevent the callback request from resuming the session of the customer (when the session ID exists in the callback request URL).

The negative aspect of this approach is that if the customer did not confirm the payment at PayPals site, an order entry would still exist in the database marked with the order status of “Preparing”. Logic exists in the osC_Order class to replace orders marked as “Preparing” when the customer visits the checkout/confirmation page multiple times (incase they wanted to add a product to their cart at the last minute) to avoid the orders database table being flooded with “Preparing” order entries.

The drawback here is the possibility of the order still being deleted (replaced) even if the customer confirmed payment at PayPals site. This can happen only if the callback request is delayed and if the customer does not hit the last continue button at PayPals site to return to the store normally, but uses the browsers back button (or a bookmarked link) to return to the store and goes through the checkout procedure again to make another order.

This situation would be rare though and have confirmed with several people directly that they had not experienced this with the official PayPal IPN payment module contribution for 2.2MS2.

These are just some examples of the technical issues involved when working on the payment modules, the checkout procedure, and again on the payment modules if one needs special requirements

An addition to the checkout procedure is a re-introduction of the credit card class that was removed a while ago. The Credit Card Types Administration Tool page is now combined with the new credit card class, where each credit card contains a perl based regular expression pattern to match against provided credit card numbers. Two new parameters are provided to the store owner to control the pattern matching behaviour, where it can be enabled or disabled for server side matching (through PHP) and on the client side with javascript.

The benefits of perl based regular expressions (preg) over posix based regular expressions (ereg) is that the same pattern can also be used for client side javascript matching.

In addition, the credit card payment modules now share a relationship with the Credit Card Types where specific credit cards can be enabled for specific payment modules.

This is shown nicely at the end of the attached movie, demonstrating the post transaction actions again with Authorize.net and PayPal.

Now that my development branch is stable again, I’ll be able to forward a pre3.0alpha3 version to the payment service providers to start the certification phase. A page on the support site will be added next week to show the progress made here, and will mention it in a blogging entry as soon as it is ready for public viewing.

Here is a second movie demonstrating the post-transaction actions of 3.0 for your viewing pleasure

oscommerce_30alpha3_post_transactions.mov

I’ve had a lot of fun the past week with the payment modules; not only working on post-transaction actions, but also on the backend API side of things and have ultimately split the way the modules are structured.

Similar to work being done on combining the catalog and administration tool files, there are now two main files to a payment module, one for the catalog side taking care of the initial transaction for a new order, and a second file for the administration tool taking care of installation, removal, and post-transaction actions. The second set of module functions are never used nor needed on the catalog side, so moving these over to the administration tool side cleans up the API and adds in an extra level of security. This also brings in a performance increase on the catalog side, but won’t be seen (hey, there’s less inclusion of code going on here )

The modules that communicate with their gateways via a HTTPS request can return the response with just one line of code:

$this->_transaction_response = $this->sendTransactionToGateway('https://test.authorize.net:443/gateway/transact.dll', $post_string);

(the sendTransactionToGateway() class method is part of osC_Payment, so it is available to all payment modules)

The cool thing about this function is that it falls back nicely onto what the server is capable of doing to make that HTTPS request. The first thing that is tried is using the native PHP functions when OpenSSL is built with the PHP installation (PHP 4.3+ feature). If this isn’t available, it falls back to using libcurl, which is “curl” built into the PHP installation, and if the server doesn’t support that, it automatically falls back to using the cURL binary file that should exist on the server.

I really do not like the cURL binary file method as the credit card details are passed to it via exec(), which can then be seen on the server via the “ps” command. If this is the case for your shared webserver, I’d recommend changing hosts if you want to accept credit cards this way, as it can be insecure.*

* Some modules are able to pass the parameters encrypted to “curl” (using exec()) but is unfortunately not common.

One problem that I wanted to tackle properly with the payment modules API update, was to take care of the cat and mouse game of the order being created after a payment request to the gateway has been made, specifically the desire and need of passing the order ID to the gateway for its intended financial transaction.

The problem here is that the order ID does not exist at this stage as the order has not been created yet. It was common to pass just a randomized value to the gateway but had no official relation to the order as this value was not being stored with the creation of the order.

With my thinking cap on, I would have liked to have solved this problem by using database sequences, similar to MySQL’s auto_increment feature, as it is not possible to neatly grab the current (or next) auto_increment value for a specific table before an insert has been performed (argh; cat and mouse game). But regardless of using auto_increment or sequences, the database tables involved of storing a new order would need to be locked so that the relationships between the tables match the order being created (ie, osc_orders and osc_orders_products; by orders_id).

And this is what the ultimate problem is: unfortunately not all hosting providers have granted their customers with LOCK privileges.

No, I did not have to ring every hosting provider in the world to ask if they provide LOCK privileges to their customers or not, but did some research and came across this problem while searching for alternative methods. One of the projects that are experiencing this problem is Drupal:

http://drupal.org/node/39460

They use the more neutral (or standard) approach of using sequences over MySQL’s auto_increment feature, and have been experiencing this problem with their releases.

As osCommerce is not yet officially database independent (although the API for it is just about there), and we are focusing on compatibility with MySQL 3.x -> 5.x with MyISAM and InnoDB tables, I took my thinking cap off and decided to keep it simple, yet still secure, to have this work on each and every MySQL database server out there (thank you MySQL drawbacks! ) and keep everyone happy.

The solution that has been implemented is to create that random order ID value, make it unique, and to store that with the order when it is created. That way there is an official relationship between an order and a transaction that can be seen. Whether or not the ID passed to the payment gateway is the order ID or another ID does not (or should not) really matter; the important thing here is that a relationship between the order and transaction exists, and prevents duplicate orders from being sent (another reason some payment gateways require an order ID to be sent to them).

This has no affect on the post-transaction actions side of things (eg, capturing an authorized transaction), as the gateways return their own transaction ID which is used to identify the transaction for post-transaction actions.

That was my thinking-cap experience for the week, and had fun finalizing the modules I got started on and starting on new ones. The certification from the payment service providers will begin next week, and look forward to the results this will bring.

The communication here to the payment service providers has been great, with all being anxious in seeing and certifying the new modules.

I will post another video of what is possible now with the post-transaction actions in the coming days, so stay tuned!