tofubl

joined 1 year ago
[–] tofubl@discuss.tchncs.de 2 points 11 months ago (2 children)

Here's a cool article I found on Nextcloud performance improvements, and connecting Redis over Unix sockets gave me a more substantial performance improvement than migrating to Postgres. Very happy I fell down this rabbit hole today.

To note if you're following the tutorial in the link above, and for people using the nextcloud:stable container together with the recommended cron container:

  • the redis configuration (host, port, password, ...) need to be set in config/config.php, as well as config/redis.config.php
  • the cron container needs to receive the same /etc/localtime and /etc/timezone volumes the app container did, as well as the volumes_from: tmp
[–] tofubl@discuss.tchncs.de 8 points 11 months ago (4 children)

Okay, did the migration just now. Everything seems a little more responsive, but I wouldn't call it way faster.

Either way, it wasn't very scary at all. For anybody coming after me:

  • add postgres container to compose file like so. I named mine "postgres", added a "postgres" volume, and added it to depends_on for app and cron
  • run migration command from nextcloud app container like any other occ command and check admin settings/system for db state: ./occ db:convert-type --password $POSTGRES_PASSWORD --all-apps pgsql $POSTGRES_USER postgres $POSTGRES_DB
  • remove old "db" container and volume and all references to it from compose file and run docker compose up -d --remove-orphans
[–] tofubl@discuss.tchncs.de 2 points 11 months ago* (last edited 11 months ago) (6 children)

Interesting. Do you remember where you read this?

The process seems simple enough. I'm on the nextcloud:stable docker image, so adding a postgres container is really easy, but it's a scary task...

[–] tofubl@discuss.tchncs.de 2 points 11 months ago* (last edited 11 months ago)

A very welcome compliment after this ordeal. Thank you! :)

The cleanest way to solve the scaling issue would probably be to go into the pdf2pic module and hack it open to accept the "pcl:fit-to-page" option that GraphicsMagick (the underlying software package doing the actual conversion from PDF to PCL) supports. (Supposing it actually does what it says. I'm not so sure about anything in printer land anymore.)

But since this whole thing is for internal documents only and the scaling can probably be estimated by choosing better values for width/height to account for printer margins I most likely won't bother.

Thanks again for suggesting Node-RED. I'm very happy with the result.

[–] tofubl@discuss.tchncs.de 2 points 11 months ago* (last edited 11 months ago) (2 children)

One deep dive into the IPP protocol, printer drivers, and specific supported formats ("what the hell is an octet-stream?!") later, I have something functional.

I'm not a JS guy, so I don't know if it's a very node-y way to do it. But except for the pages coming out a little scaled this works:

Edit to briefly explain what this does: The mail node ingests unread emails and passes them forward to a function node that checks for pdf attachments. Those are passed forward one by one to a function node that takes care of converting the pdf to pcl (my printer supposedly knows how to handle pdf 1.7 but doesn't, so I had to resort to this) and passes it to the IPP node as a data buffer .

spoiler[ { "id": "86082bed0ed29155", "type": "IPPrint", "z": "08cc7c15668f2f65", "name": "Print to Network", "IP": "10.10.0.19:631/ipp/print", "JOB_name": "default_job_name", "authuser": "", "authpassword": "", "authcheck": "", "x": 780, "y": 120, "wires": [ [ "b1a8b92ede8a78a5" ] ] }, { "id": "b1a8b92ede8a78a5", "type": "debug", "z": "08cc7c15668f2f65", "name": "debug 1", "active": true, "tosidebar": true, "console": false, "tostatus": false, "complete": "true", "targetType": "full", "statusVal": "", "statusType": "auto", "x": 960, "y": 120, "wires": [] }, { "id": "1fb7fc3843af8e05", "type": "function", "z": "08cc7c15668f2f65", "name": "Convert PDF to PCL", "func": "const baseOptions = {\n width: 2480,\n height: 3508,\n density: 300,\n preserveAspectRatio: true,\n format: 'PCL'\n};\n\n// prepare conversion function\nconst convert = await pdf2pic.fromBuffer(msg.payload, baseOptions);\n// set additional conversion options\nconvert.setGM\n// bulk convert the whole input buffer to a pcl output buffer\nconvert.bulk(-1, {responseType: \"buffer\"}).then((outputs)=>{\n // pass each page on to the printer\n outputs.forEach((output, index) => {\n msg.payload = output.buffer;\n if(msg.topic){\n let jobname = msg.topic\n }\n msg.JOBName = (msg.topic ? msg.topic : \"Mailjob\").concat(\" \", index+1, \"/\", outputs.length);\n msg.docFormat = \"application/octet-stream\";\n node.send(msg);\n });\n});\n", "outputs": 1, "timeout": 0, "noerr": 0, "initialize": "", "finalize": "", "libs": [ { "var": "pdf2pic", "module": "pdf2pic" } ], "x": 560, "y": 120, "wires": [ [ "86082bed0ed29155" ] ] }, { "id": "224317e794da3cb5", "type": "e-mail in", "z": "08cc7c15668f2f65", "name": "Mail ingest", "protocol": "IMAP", "server": "test.mail.com", "useSSL": false, "autotls": "always", "port": "143", "authtype": "BASIC", "saslformat": true, "token": "oauth2Response.access_token", "box": "INBOX", "disposition": "Read", "criteria": "UNSEEN", "repeat": "300", "fetch": "auto", "inputs": 0, "x": 160, "y": 80, "wires": [ [ "2fedaa18c9394b89" ] ] }, { "id": "2fedaa18c9394b89", "type": "function", "z": "08cc7c15668f2f65", "name": "PDF Filter", "func": "if(msg.attachments.length > 0){\n msg.attachments.forEach(function(attachment) {\n if(attachment.contentType == \"application/pdf\"){\n var newmsg = {};\n newmsg.topic = msg.topic;\n newmsg.filename = attachment.filename;\n newmsg.payload = attachment.content;\n node.send(newmsg);\n }\n });\n}", "outputs": 1, "timeout": 0, "noerr": 0, "initialize": "", "finalize": "", "libs": [], "x": 360, "y": 80, "wires": [ [ "1fb7fc3843af8e05" ] ] }, { "id": "3d8f7ad8e6b56d46", "type": "inject", "z": "08cc7c15668f2f65", "name": "Test print", "props": [ { "p": "payload" }, { "p": "topic", "vt": "str" } ], "repeat": "", "crontab": "", "once": false, "onceDelay": 0.1, "topic": "", "payload": "", "payloadType": "date", "x": 160, "y": 160, "wires": [ [ "683ed6fe7c10540d" ] ] }, { "id": "683ed6fe7c10540d", "type": "file in", "z": "08cc7c15668f2f65", "name": "", "filename": "/data/2517.pdf", "filenameType": "str", "format": "", "chunk": false, "sendError": false, "encoding": "none", "allProps": false, "x": 340, "y": 160, "wires": [ [ "1fb7fc3843af8e05" ] ] } ]

[–] tofubl@discuss.tchncs.de 1 points 1 year ago

I do actually use Node-RED myself for my zigbee home automation. Funny how it never crossed my mind to pick it up for this. It's perfect as I want a non-programmer to be able to maintain it.

I doubt my solution will be much more than ingest and print, but I'll report back. Still fighting to get that hp p3015 to do something over IPP...

[–] tofubl@discuss.tchncs.de 2 points 1 year ago* (last edited 1 year ago) (6 children)

Now this is an idea I like. Thank you so much, I never would have considered Node-RED for this application!

[–] tofubl@discuss.tchncs.de 1 points 1 year ago

Thank you for the suggestion. Not exactly what I'm looking for, sadly.

[–] tofubl@discuss.tchncs.de 3 points 1 year ago

Gotcha. Thanks anyway. 🙂

You're right about printing not being all that, but certain situations (my situation) call for it, at least for the time being.

This is for manufacturing orders in a low tech environment, and at this point in time I need to stick with a paper trail (so I can get rid of it in the future.)

[–] tofubl@discuss.tchncs.de 1 points 1 year ago (2 children)

Thanks for the reply. I don't think I fully understand what you mean.

What I would like to have is a service running on a server, fetching and printing mails. Could you please elaborate how your idea would work?

[–] tofubl@discuss.tchncs.de 1 points 1 year ago

Thanks for the idea. It's not quite what I was looking for, but I might look into it more if I don't find anything better.

From the looks of it, it requires python scripting anyway, so there isn't a big advantage to fetching the mail with python and imaplib directly.

[–] tofubl@discuss.tchncs.de 5 points 1 year ago* (last edited 1 year ago) (3 children)

The killer feature for me is my networked scanner scanning directly to the paperless consume samba share and the documents just popping up in the inbox fully OCRd and pre-categorised. Pretty magical.

NB, the docs make it sound like a proper DB is optional, but it's really not. Performance was iffy for me with sqlite but is rock solid with Postgres.

view more: ‹ prev next ›