Converting Large XLS, XLSX files to CSV using Python
I was in troubles to convert big xlsx files in PHP because there is no such a library that could make it as good as possible. What I’ve found out were libraries that could manage xls / xlsx files but it took a lot of time and performance was horrible.
So, not spending too much time to think about it I’ve figured out to leave PHP and try to make it in Python. What I wanted to achive was converting xls files into csv and handle it in PHP.
In Python we got xlrd library that enables us to make this “crazy” thing. Here it is: https://pypi.python.org/pypi/xlrd. You can install it using pip, such like:
1 |
$ pip install xlrd |
But for my case it was better to use VirtualENV which is better solution when someday you’d like to move your software somewhere and you forgot about installing all those stuff. However, each way you chose would be fine till it’s working.
After installing required items all you need is to run the script that converts xls into csv. I have already written simple script that can make it. Please find code below and save it to your local system setting the proper permissions right (chmod +x filename).
convert_xls.py
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 |
import logging import time import traceback import xlrd import csv import sys import re logging.basicConfig(level=logging.INFO, format='%(asctime)s %(message)s') def csv_from_excel(): xls = sys.argv[1] target = sys.argv[2] logging.info("Start converting: From '" + xls + "' to '" + target + "'. ") try: start_time = time.time() wb = xlrd.open_workbook(xls) sh = wb.sheet_by_index(0) csvFile = open(target, 'wb') wr = csv.writer(csvFile, quoting=csv.QUOTE_ALL) for row in xrange(sh.nrows): rowValues = sh.row_values(row) newValues = [] for s in rowValues: if isinstance(s, unicode): strValue = (str(s.encode("utf-8"))) else: strValue = (str(s)) isInt = bool(re.match("^([0-9]+)\.0$", strValue)) if isInt: strValue = int(float(strValue)) else: isFloat = bool(re.match("^([0-9]+)\.([0-9]+)$", strValue)) isLong = bool(re.match("^([0-9]+)\.([0-9]+)e\+([0-9]+)$", strValue)) if isFloat: strValue = float(strValue) if isLong: strValue = int(float(strValue)) newValues.append(strValue) wr.writerow(newValues) csvFile.close() logging.info("Finished in %s seconds", time.time() - start_time) except Exception as e: print (str(e) + " " + traceback.format_exc()) csv_from_excel() |
If you have a python script created in your local filesystem the next step would be manage it from PHP. And this is most simplest solution. Using exec, system, or shell_exec solves the problem.
1 2 3 |
<?php exec(‘/usr/bin/python convert_xls.py input.xls out.csv’); ?> |
OR in command line.
1 |
$ /usr/bin/python convert_xls.py input.xls out.csv |
I know that solution is not really clean but it works. To have it more elegant I would suggest to create some kind of service in Python that handles it in a good way. I will think about it …
Note: I have also fixed issue with Long numbers in XLS.
Update
If you need to parse big XLS, XLSX files I would suggest not to make in in PHP. Just parse this using JAVA where PHP would be used only for UI.
Any questions, let me know. Please find comment form below.
Drupal, UberCart – How to return / get back the cart from database
I’ve tried to found the solution how to revert the cart that is already in the database but as always I didn’t found any good solution though. I was looking the files which can help me to regenerate it and finally I’ve found it! Below you got the code that get any cart from the db and populate it.
1 2 3 4 5 6 7 8 9 10 |
function revertCart($cartId) { $items = uc_cart_get_contents($cartId, 'rebuild'); if (! empty($items)) { foreach ($items as $item) { uc_cart_add_item($item->nid, $item->qty, $item->data); } } } // revertCart('the ID from uc_cart_products |
Enjoy!
How to Change the Original Price in Prestashop
That’s a really useful snipet for anybody who wants to change the original price to the new one.
Solution for this is very easy …
Ok .. here it is:
Product.php, line 1686
1 2 3 4 5 6 |
public static function getPriceStatic($id_product, $usetax = true, $id_product_attribute = NULL, $decimals = 6, $divisor = NULL, $only_reduc = false, $usereduc = true, $quantity = 1, $forceAssociatedTax = false, $id_customer = NULL, $id_cart = NULL, $id_address = NULL, &$specificPriceOutput = NULL, $with_ecotax = TRUE) { // .... find some space before `return` $my_new_price = Warehouse::getPriceByDay(); return $my_new_price; |
How to Add a New Custom Field in Prestashop
I’ve figured out how to add new field to database using Prestashop (1.4.3, but I think for the newer versions would be the same). This solution gives you new field in database and you will be able to edit / save data to this field in your admin panel, for my example I needed a field which was called “number of photos”. Below you can see how it works.
Read the full article »
Getting Cart From Prestashop
If you have a problem with getting cart from prestashop that would be interesting for you.
I had a huge problem with Prestashop (version 1.4.3) because I didn’t know how exactly I should get cart from Prestashop to the WordPress or somewhere else. However I spent some time to resolve that issue and I wrote that:
First of all we should get session data from Pretashop:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 |
require_once dirname(__FILE__) . '/config/config.inc.php' ; require_once(dirname(__FILE__) . '/config/settings.inc.php'); require_once dirname(__FILE__) . '/init.php' ; // this initializes the Cookie singleton, which is available in any script function getAllCookies(){ global $cookie ; // var_dump($cookie); return $cookie; } $cookie = getAllCookies(); |
As you see you should add some lines from Prestashop configuration and then you should create a function in which you have cart ID. And then you can easily get all data from the cart:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 |
$query = "SELECT p.id_product, pl.name, p.quantity, pp.price, p.id_product_attribute, pp.price * cu.conversion_rate AS orderprice, cu.sign, cu.format FROM ps_cart c LEFT JOIN ps_cart_product p ON (p.id_cart = c.id_cart) LEFT JOIN ps_product pp ON (pp.id_product = p.id_product) LEFT JOIN ps_product_lang pl ON (pl.id_product = p.id_product) LEFT JOIN ps_currency cu ON (cu.id_currency = c.id_currency) WHERE c.id_cart = " . (int) $cookie->id_cart . " AND pl.id_lang = 1 ORDER BY p.date_add ASC"; $link1 = mysql_connect(_DB_SERVER_, _DB_USER_, _DB_PASSWD_); if ($link1) { $db_selected = @mysql_select_db(_DB_NAME_, $link1); $res = mysql_query($query); $arrProducts = array(); while ($row = mysql_fetch_assoc($res)) { $arrProducts[] = $row; } var_dump($arrProducts); // all products from session cart } mysql_close($link1); |
(Sorry for quality of code but I had to wrote this as quick as possible)
And that’s all