Day changed to 2022-05-01
[00:03] jfw: rodolfo: this urllib2 snippet for example even does a POST upload without any explicit socket operations
[00:08] jfw: oh hey, it's May.
[00:28] rodolfo: Understood about the technical questions. I'll try to figure out most things on my own, if I feel like on a dead end, I'll let you know.
[00:29] rodolfo: The list is great, I'll do my best best in getting to know those new to me.
[00:38] rodolfo: Regarding the socket config, I think I misread the initial docs since it was pointing out that urllib/urllib2 requires the *sockets* module. Both urllib versions are indeed high-level interfaces. Sorry about that
[00:49] rodolfo: jfw: Regarding an external server, I was thinking about using something like Heroku or Firebase (hypothetically) where I could set up a Python3 environment and make use of modules that facilitate the logic (ie receiving an input, doing inference and outputting 'the prediction').
[00:49] rodolfo:
[00:49] rodolfo: Then, exposing an endpoint from that instance so the input from the VM env will return the prediction. Nothing complex.
[00:49] rodolfo:
[00:49] rodolfo: Makes sense?
[00:50] rodolfo: Not sure about potential latency issues, but if so then we could keep everything of the VM by making use of the native functionalities, as I mentioned before
[00:54] jfw: rodolfo: it sounds indecisive possibly; like, you're trying to use two different systems so as to get the benefits of each, but it risks that you just end up paying the costs of each. we may need some clarification on the strategic perspective. dorion, any thoughts?
[00:56] jfw: perhaps it does make sense, as a first step from what you have now, just moving the more ...front-end parts of the back-end onto gales
[00:58] jfw: but if there isn't a longer term goal of getting the whole thing moved, it's less clear to me what would be the benefit to your project of using gales at all
[01:00] jfw: or more generally, of using a jwrd-hosted server if it's going to depend on 'the cloud' anyway
[01:09] rodolfo: Yeah, it is subject of further discussion. However, the only reason why I bring it up is just to save some time. I know no one has mentioned anything regarding deadlines but if time is not an issue, then I can dedicate whatever it takes to make it work on the current server.
[01:16] jfw: heh, well it might not make sense for *you* to allow unlimited time for the project, but I won't be the judge of that. as far as keeping the current VM online, the costs are pretty minimal because we'll be using its host server for other things too. as far as my development time, the idea is that it's for things that align with the direction we were going with Gales anyway, so again not a high
[01:16] jfw: marginal cost.
[01:19] jfw: that said, we do need to keep the lights on, we can't offer free things indefinitely and the point of this project as I understand is ultimately to get revenue; so there's going to be some implicit deadline at the very least.
[01:22] jfw: "implicit" as in the situation where there seems to be plenty of time until all of a sudden there isn't; and as yet I can't say where that point is but so far it's still in the "plenty of time".
[01:31] jfw: I could certainly hear an argument to minimizing the up-front technical wrangling, focusing on getting a sellable product first of all, then using the cash flow to support & justify backend optimization, as long as the current technical decisions aren't painting us into a future corner.
[01:47] rodolfo: Yeah, I think we are pretty aligned in terms of focusing on the goal which is the production of a tool that can be provided in exchange of an inflow in whatever currency expression. I do want to establish some time parameters, nothing that could become counter-productive but just for my overall mental health is good not to procrastinate. Let me figure out a few things and I will communicate you what these time parameters could look like.
[02:01] jfw: sounds good.
[18:05] rodolfo: Hi. While I figure out a set of deadlines, I'll share with you how I'm coming along with overall experience of translate the logic from Py3 to the VMs Py2 reality.
[18:08] rodolfo: For now, I'll do the data preprocessing on my local computer, this will result in a X by X array which then I will sftp to the VM. Then, using the native modules, I will write the necessary functions for the regression process that will later result in an input method for inference.
[18:09] rodolfo: X by X, N by N.. you get the point
[18:22] dorion: http://fixpoint.welshcomputing.com/2022/jwrd-logs-for-May-2022/#4069 -- I agree with you jfw. we went down this path of using Gales becasue rodolfo said his tool is portable to python 2. rodolfo, I think an important step in the planning is to document the parts of your tool that are python 3 only. e.g. are you using libaries that only exist in python 3 ?
[18:22] sourcerer: 2022-05-01 00:54:09 (#jwrd) jfw: rodolfo: it sounds indecisive possibly; like, you're trying to use two different systems so as to get the benefits of each, but it risks that you just end up paying the costs of each. we may need some clarification on the strategic perspective. dorion, any thoughts?
[18:26] rodolfo: @dorion: hi, yes but "tropicalization" is possible with creativity and effort hehehe
[18:43] dorion: not sure what you mean by tropicalization ? rewriting the python3 things in python2 ?
[18:46] rodolfo: For example, I'm solving a workaround for the One Hot Encoding method. This technique comes out of the box in sklearn, but jfw mentioned that the current Python 2 set up might not be suitable for numpy computation and sklearn relies on numpy. Having said that, there are options that not require any "magical solution" from Py3 modules, but would return the same result.
[18:48] rodolfo: It's not just translation, but some kind of reingeneering hehehe
[18:48] rodolfo: But that's OK, I was expecting this level of challenge
[18:50] rodolfo: With the Py2 native modules I have enough to do it, from what I can see
[18:58] dorion: ok, thanks for clarifying.
[19:00] rodolfo: Sorry for not bein explicitly clear. Maybe you guys were thinking about something like print() vs print kind of translation jaja
[19:35] jfw: well, let me expand re numpy, noting that I haven't really used it and am not really up to speed on the details of Machine Learning otherwise. my understanding is that it's basically a SIMD layer for Python, giving it adequate performance for large-scale numeric computations by adding homogeneous array data types and bulk operations on them. thus, as an interpreter extension, it's not going to be
[19:35] jfw: as simple as 'pip install numpy' or whatnot on Gales, unless it has a pure-python fallback though that would seem to defeat the purpose.
[19:39] jfw: it certainly existed back before the python3 push, so it's probably doable; though as yet we don't have experience with adding these third-party extensions into the python build.
[19:41] jfw: but, if we can do without for now, we may find that the performance isn't actually a problem at our current scale.
[19:42] jfw: my suggestion to rodolfo was just to try it and document what happens, For Science, even though I'm not expecting it to Just Work.
[19:43] rodolfo: Understood my friend
[19:46] jfw: rodolfo, have you done much work with python internals (eg the extension or embedding APIs, or just patching or compiling it)?
[19:48] jfw: or are you interested in such, since you're relying on python after all?
[19:53] rodolfo: Hi. I have experience with native Py modules but, since during work "time is money" I do have to rely on frameworks that ease the direct use of the native functionalities. But, overall, my work is about solving problems whatever it takes and I'm managing myself under the same premise within the current scenario.
[19:53] rodolfo: Fear not, I have everything under control
[19:53] rodolfo: =)
[19:54] rodolfo: Update: data is already on the VM
[19:56] jfw: heh, I wasn't afraid, merely exploring possibilities.
[19:59] jfw: I was asking about internals as in the C level, not merely the standard library which is what I guess you mean by 'native'.
[20:21] jfw: regarding frameworks, I'd say there's two ways they end up being used: one, as a form of leverage, when used by a skilled hand to increase productivity; two, as a crutch, when used out of ignorance or laziness to use simpler tools. In either case you pay the costs of adopting the framework, or worse, someone else down the line has to pay them, when it breaks or gets in the way or otherwise needs
[20:21] jfw: maintenance.
[20:23] jfw: that's why I'm not in favor of "teaching kids to code using python, it's so friendly"; let them start with machine code and work their way up to the power tools.
[20:43] rodolfo: Yeah, I'm considering some C/C++ possibilities but right now I might stay in the Py realm: after having all the data at the VM, I accomplished setting up some udf/modules to read the data accordingly.
[20:44] rodolfo: Next step - - > regression process
[20:48] rodolfo: To make my story a little bit sad, I never had the opportunity to actually learn any of this until I was 34 year-old. However, my previous experience was with extensive data analysis so it wasn't completely difficult for me to understand some concepts. My "tech lingo" comes from reading informal information sources, sorry if by communicating any message I don't do it properly. I'll try to do my best
Day changed to 2022-05-02
[00:24] jfw: rodolfo: hey no worries, you've got to start somewhere; and as a 32 year old I certainly hope 34 isn't the cutoff for being allowed to pick up new things!
[00:26] jfw: sounds like good progress on getting set up with the data. are you keeping notes on your steps, at minimum for yourself so you can repeat them later?
[00:55] rodolfo: Yeah man, I'm 37 yo and still learning a bit of everything
[00:56] rodolfo: Yes sir, I can replicate everything I'm currently doing
[18:20] rodolfo: UPDATE: I'm working on the calculations required for factor sensibilities.
[18:24] rodolfo: @jwrd could you explain how to use the "gports" method for installing add-ons?
[18:24] rodolfo: @jfw
[18:27] rodolfo: https://usercontent.irccloud-cdn.com/file/jKXTvRql/Screenshot_20220502_132659.jpg
[18:36] rodolfo: I think I can use pip directly in the terminal, but, like I mentioned before, PYTHONHOME (or similar Py path variable) is not defined yet. Is the Python Path = /gales/pkg/python-0.3-2.7.13/bin/python?
[18:37] rodolfo: Please confirm, thanks!
[18:59] jfw: rodolfo: I'd say you need to focus more on the "whats" and "whys" before jumping to the "how tos", especially since you're in an unfamiliar environment; otherwise how do you know that what you think you want to do is actually what's needed?
[19:07] jfw: to (perhaps annoyingly) answer exactly what was asked though, you install gports by obtaining the Gales repository and distfiles, entering the relevant subdirectory of "gports", running "gbuild" (preferably as the "build" user), then "gpkg-install -f" on the resulting package file; as explained in the PORTS file in the same repository. The path you show is indeed the concrete path to the python
[19:07] jfw: executable, but it's unclear what you mean by Python Path or why PYTHONHOME is necessary.
[19:07] rodolfo: I know this whole interaction with Gales is kind of an experiment (from my side), however I do believe some ease of use could become handy in order to replicate (sooner than later) what I did on a regular env. My point is that dealing with Gales shouldn't become a struggle for at least getting the basic functionalities (ie installing Py modules via pip). I don't think it adds any value to come up with my version of, for example, performing a t-distribution
[19:07] rodolfo: in order to find P-values for each factor. I'm not sure if I'm being clear enough, perhaps I can give RD a call in order to contextualize my comments.
[19:08] jfw: rodolfo: what would you say "pip" is for?
[19:09] rodolfo: The Py repository of packages/libraries
[19:09] jfw: well PyPI is not the same as "pip" though, is it?
[19:09] rodolfo: Let's say I need to install scipy, how do you recommend doing so
[19:10] rodolfo: It could be pip, or whatever other method. That's why I mentioned the gports you were referring to earlier
[19:12] rodolfo: It could be even from another repository
[19:13] jfw: well, I'd hazard a definition of pip as "a tool that automatically imports third-party code into your system, in an attempt to support the pretense that such code is external to your codebase and someone else's problem"
[19:13] jfw: in other words: in my view it's automating the wrong thing
[19:14] rodolfo: It's OK, let's say I just need the miracle (installing X library from a repository)
[19:14] jfw: so if I wanted to run scipy, my approach would be to look up scipy itself, find how to obtain it, check the docs, what versions are available, how to install etc.
[19:14] rodolfo: I understand
[19:16] rodolfo: In conclusion, there is no automatic way to get/install a library from a repository (in the exact same way I could achieve it with pip or similar methods). It should be done manually. Correct?
[19:17] jfw: I can't as yet say why pip isn't working for you; I didn't intentionally kill it or anything
[19:17] jfw: just that it's not very interesting to me, I suppose.
[19:17] rodolfo: Not saying that, I just want to have a clear view of the situation
[19:18] jfw: but I expect that doing things manually at least for the initial exploration would be enlightening
[19:19] jfw: meanwhile I've got the details on the apparent pip failure: http://welshcomputing.com/paste/suaatqhatu
[19:21] jfw: the exec_prefix/PYTHONHOME thing looks like a red herring - I don't expect there are any "platform dependent libraries" to load since it's a static build, perhaps it's a python or packaging bug - but the real problem is simply the "No module named pip". It's not installed!
[19:23] jfw: I expect "setuptools" at least will be required to install most python packages these days, even manually; and there is a gport for that, which I can install for you if that's what's needed.
[19:36] rodolfo: The "pip situation" can potentially be fixed with a wheel called "ensurepip" which is currently in the VM
[19:37] rodolfo: Let me explore
[19:43] jfw: miracles on wheels, that's what the market has come to expect these days.
[20:04] jfw: meanwhile in a total random coincidence, from what the market actually delivers: "We are working on restoring our services following a network outage. At the moment we are unable to process any orders. We apologize for the inconvenience." - major electronics components distributor
[20:31] rodolfo: Chaos is everywhere
Day changed to 2022-05-05
[16:50] dorion: hey rodolfo, for the mvp, we're thinking it's probably best to just stick with flask for the front end because a) that's what you already have b) apache isn't ready yet on gales and c) using php and apache is going to require some deep thinking on the architecture.. not that deep thinking is something to be avoided, just that it looks like a premature optimization and sticking with flask means the
[16:50] dorion: ability to cashflow it sooner.
[17:05] jfw: ah thanks for refreshing that point dorion. to expand a bit, it sounds like the current implementation does much of the work on-demand in python, so decoupling the 'data parts' from the 'web parts' isn't straightforward.
[17:06] jfw: and there's many input parameters so not well suited to pre-generating static html.
[17:07] jfw: the work to be done on the Gales porting front then would be getting the various numeric & other such libraries working
[17:08] jfw: and on the business logic side, implementing some type of subscriber authentication.
[18:30] jfw: http://fixpoint.welshcomputing.com/2022/the-passive-threads-that-would-like-to-be-given-some-blocks/
[21:39] jfw: aaand http://fixpoint.welshcomputing.com/2022/two-patches-and-a-new-fetch-bitcoind-release/
[21:42] jfw: (the article that was supposed to come first!)
[21:58] dorion: jfw, nice !
[21:58] dorion: hello donald, who might you be ?
Day changed to 2022-05-07
[01:26] jfw: I've added a browser-accessible gateway to the pastebin service, presented as the usual text box form input; so now it should be usable indeed from just about anywhere.
[01:28] jfw: surprisingly the php sockets functions weren't enabled by default, so I added --enable-sockets to the ./configure list in the php gport.
[01:30] jfw: (they're used for connecting to the backend which is the same plain TCP responder on port 4.)
Day changed to 2022-05-08
[12:58] caai: jfw: Could you verify that I am creating the run script for /etc/svc/bitcoind correctly? My first estimation is:
[12:58] caai: #!/bin/sh
[12:59] caai: exec bitcoind /home/user/.bitcoind
[20:15] donald: Justin - also known as donald or jwm
[20:17] donald: hexchat is a bit funky, didnt remember my original login - ubuntu is special!
[20:18] jfw: indeed, though I suspect the 'snap' thing that wanted to be cooler than apt contributed to the tangle.
[20:22] jfw: caai: good start, but bitcoind doesn't take a directory argument like that, does it? I'm guessing the reason you wanted to put it there is valid though
[20:23] jfw: if so, what do you think the underlying problem is?
[20:34] caai: well, in order to start bitcoind, no argument is added, so I understand my logical misstep.
[20:35] caai: On the other hand, bitcoind must be executed within the .bitcoind directory, therefore:
[20:35] caai: #!/bin/sh
[20:36] caai: exec /home/user/.bitcoind/bitcoind
[20:37] caai: ?
[20:37] caai: however, there is no 'file' named bitcoind in the .bitcoind directory
Day changed to 2022-05-09
[02:18] caai: which bitcoind indicates: /local/bin/bitcoind
[04:33] jfw: caai: that would be one sense of "executed within" I suppose - where the executable itself is located in that directory; though perhaps you have in mind the other sense, of being the working directory in which the process is run. (recall program is to process something like what dna is to individual organism)
[04:39] jfw: but, I don't believe it has to be executed in any particular working directory. It does however have to be run as the correct user, and since the service supervisors run initially as root, perhaps this is at the ...root of your trouble?
[04:40] jfw: if you want to set the working directory, you'd add a "cd" command; it's simply scripting the same commands you'd run interactively at the terminal.
[04:42] jfw: to set the user, perhaps you could mess around with "su" but daemontools includes a more handy thing for the purpose, setuidgid
[04:44] jfw: which gets us to something like: "exec setuidgid user /local/bin/bitcoind"
[04:44] jfw: next, I'd redirect the command's standard error into its standard output so they both get captured by the log sub-service, if any: "exec setuidgid user /local/bin/bitcoind 2>&1"
[04:47] jfw: finally, I see in my own run script I'm also setting up the HOME environment variable, presumably because setuidgid is too narrowly specified to take care of such conveniences, and bitcoind needs it to find the correct home directory.
[04:49] jfw: I'm mildly curious to see what happens without it though, so I'll leave that part as an exercise :)
[19:43] dorion: http://fixpoint.welshcomputing.com/2022/jwrd-logs-for-May-2022/#4170 -- hey, welcome back.
[19:43] sourcerer: 2022-05-08 20:15:08 (#jwrd) donald: Justin - also known as donald or jwm
Day changed to 2022-05-10
[11:49] caai: jfw: I created the 2 lines in the 'run' file as inidicated: "exec setuidgid user /local/bin/bitcoind" and "exec setuidgid user /local/bin/bitcoind 2>&1"
[11:51] caai: next, 'chmod +x /etc/svc/bitcoind/run'
[11:52] caai: then, svc -u /services/bitcoind
[11:53] caai: (perhaps that was unnecessary and it would be up by default)
[11:59] caai: finally, upon running 'svstat /services/bitcoind', I see that the pid changes each time I run it and that the service is up but only for 1 second on average, thus, indicating that it is going up and down
[17:03] jfw: caai: I didn't mean two separate lines, rather was building up the one line by parts, so just keep the second. then, you're correct, it's starting then immediately exiting; so to find out why, you have two log files to check: the standard output/error captured in /var/log/bitcoind/current, or the usual ~/.bitcoin/debug.log.
Day changed to 2022-05-13
[14:00] caai: jfw: I see. I have reduced the script to the second line
[14:07] caai: next, I stopped the node via 'kill -9', then i 'tail -f ~/.bitcoind/debug.log' | 'svc -u /services/bitcoind'
[14:11] caai: 'tail -f ~/.bitcoin/debug.log'*
[14:12] caai: as I observe result of (tail -f ~/.bitcoin/debug.log), I see no output
[14:13] caai: when I 'ls /var/log/' the output is klog and sshd. I see no directory for bitcoind
[20:02] caai: cleary, i am missing something .....
Day changed to 2022-05-14
[05:08] caai: clearly*
[14:34] rodolfo: Good day friends!
[14:35] rodolfo: @dorion today I'm taking my FRM level 1 exam, I'll be out around 2:30 pm
[15:17] rodolfo: Man, I really live at a fucking jungle.
[15:18] rodolfo: The testing center forgot to open their facilities today.
Day changed to 2022-05-16
[18:48] jfw: caai: I'll guess that there's no log sub-service defined; you can check this via 'svstat /service/bitcoind/log'. if that's the case, you can create one using the generic/default log service from gales: mkdir -p /etc/svc/bitcoind/log ; ln -s /etc/svc.defs/multilog /etc/svc/bitcoind/log/run
[18:48] jfw: once that comes up, /var/log/bitcoind should be created and start capturing whatever's going wrong.
[18:49] jfw: rodolfo: ouch, sounds very LatAm/tropical indeed.
[18:50] jfw: that's a Financial Risk Manager exam I take it
Day changed to 2022-05-17
[13:24] caai: jfw: upon running 'svstat /service/bitcoind/log', I saw that there was no log file. I created one running 'mkdir -p /etc/svc/bitcoind/log ; ln -s /etc/svc.defs/multilog /etc/svc/bitcoind/log/run'
[14:31] caai: I then ran 'svc -u /service/bitcoind'. After that, I ran 'ls /var/log/'. The 2 directories displayed are klog and sshd, but no bitcoind file appears
[14:33] caai: I will continue to investigate this matter tomorrow
Day changed to 2022-05-18
[04:47] jfw: caai: from some testing and reading between the lines of the svscan docs it seems this case of a log service being added after initial supervisor startup isn't really provided for. thus to remedy it, short of rebooting the machine, we'll have to fully disable the service then re-trigger it.
[04:48] jfw: 1) rm /service/bitcoind [to prevent svscan from restarting the un-logged supervisor]
[04:48] jfw: 2) svc -x /etc/svc/bitcoind [to signal the un-logged supervisor to bring down the daemon then exit itself]
[04:49] jfw: 3) ln -s /etc/svc/bitcoind /service/ [to re-enable so svscan will start the now-logged supervisor]
[04:53] jfw: within 5 seconds, 'svstat /service/bitcoind/log' should indicate up and the /var/log/bitcoind directory should be created.
[14:49] caai: jfw: I followed all of the above steps and the /var/log/bitcoind directory was created. Thank you! I will now search in the log to discover what I can
[14:55] dorion: caai, nice !
[18:18] jfw wonders if "work out how to create your own daemontools service for bitcoind" was too much for a homework assignment. perhaps just too much for a *first* unguided one in that direction
Day changed to 2022-05-20
[11:22] caai: jfw: when i 'tail /var/log/bitboind/current' the most relevant information i see in the error message is:
[11:23] caai: 'tail /var/log/bitcoind/current'*
[11:30] caai: [what() boost::filesystem::create_directory: Permission denied: "/.bitcoin"]
[11:58] caai: I imagine this error is due to not having file system access due to not having user privileges. Since all of the service files have a uid and gid of 'root', that seems odd because root has unlimited privileges.
[12:05] caai: however, in the script '/etc/svc/bitcoind/run' setuidgid is 'user'
[12:17] caai: I setuidgid to 'root' and it seems to be executing properly now
[12:25] caai: it is not running bitcoind yet, but the logs are showing some configurations that i need to change in /.bitcoin/bitcoin.conf
[12:28] dorion: caai, what's in /.bitcoin ? compare that with /home/user/.bitcoin
[12:32] dorion: http://fixpoint.welshcomputing.com/2022/jwrd-logs-for-May-2022/#4229 -- I'm not surprised that it's executing now, but suspect it's not proper.
[12:32] sourcerer: 2022-05-20 12:17:15 (#jwrd) caai: I setuidgid to 'root' and it seems to be executing properly now
[12:38] dorion: might have something to do with : http://fixpoint.welshcomputing.com/2022/jwrd-logs-for-May-2022/#4188 http://fixpoint.welshcomputing.com/2022/jwrd-logs-for-May-2022/#4189
[12:38] sourcerer: 2022-05-09 04:47:32 (#jwrd) jfw: finally, I see in my own run script I'm also setting up the HOME environment variable, presumably because setuidgid is too narrowly specified to take care of such conveniences, and bitcoind needs it to find the correct home directory.
[12:38] sourcerer: 2022-05-09 04:49:40 (#jwrd) jfw: I'm mildly curious to see what happens without it though, so I'll leave that part as an exercise :)
[16:55] jfw: caai: dorion has it; we don't want to run it with unlimited privileges but continue running as 'user' as before, that's why the setuidgid. so the problem is that it's trying to use /.bitcoin for data dir rather than the correct /home/user/.bitcoin that we already set up, and that's because of the missing HOME environment variable.
Day changed to 2022-05-21
[14:28] caai: I see. I have edited the script to 'setuidgid user'. I am now researching how to set environment variables.
[22:42] jfw: Sound the trumpets, beat the drums...
[22:42] jfw: http://fixpoint.welshcomputing.com/2022/sending-buffer-overflows-fixed-in-bitcoind-and-other-cleanups/
Day changed to 2022-05-22
[00:11] dorion: jfw, well done, quite the enjoyable read there. I liked the inclusion of the code in the article for this one. You added many comments in the code to so there were a few levels of commentary going on.
[04:53] jfw: thanks dorion, indeed there were multiple levels there.
[05:26] jfw: ah, http://billymg.com/2022/01/building-trb-on-a-2022-vintage-musl-gentoo/ was the ref I was looking for at http://fixpoint.welshcomputing.com/2022/jwrd-logs-for-Apr-2022/#3850
[05:26] sourcerer: 2022-04-18 17:05:32 (#jwrd) jfw: http://welshcomputing.com/paste/hhz52895bm << error encountered in building bitcoind on a stock ubuntu with gcc "10.3.0", coming from improper redefinition of __atomic_compare_exchange. I swear I'd seen this before but haven't turned up any discussion.
[05:27] jfw: still need to decide on and integrate a proper solution to these.
[05:35] jfw: that, and/or do a recipe for building in a Gales chroot.
[05:39] jfw: but for the 'let me just try this out on ubuntu without bootstrapping a whole new system' use case that might be overkill.
[05:40] jfw: a low priority perhaps, but the code is technically being stupid and ideally would get fixed.
Day changed to 2022-05-23
[18:57] caai: jfw/dorion: shall we have the next management meeting on: May 31 - 23:00 UTC? | I think that I have to make more progress with my assignments before the scheduling the next Operator Meeting
[18:57] jfw: caai, that should work for me
Day changed to 2022-05-25
[12:41] caai: jfw: I believe I am aproximating a line with the correct syntax to set home environment variable. I have tried numerous variations of "export PATH=etc.." to no avail
[12:44] caai: At the moment, I am reading the man page for ksh to learn what I can
[13:46] dorion: caai, you're trying to set HOME rather than PATH, right ?
[17:45] jfw: ^
[17:46] jfw: as strong a move as reading the ksh man page may be, looks like this more basic confusion.
Day changed to 2022-05-27
[13:21] caai: alright. my next guess was to try "export HOME=/home/user", which was unsuccessful. After that, I tried "export HOME=/home/user:$HOME", which was also unsuccessful
[17:59] jfw: caai: the first one there should be correct so perhaps something else is wrong; maybe paste the full "run" script you came up with?
Day changed to 2022-05-28
[14:30] caai: alright. here is the full 'run' script that I came up with:
[14:31] caai: #!/bin/sh
[14:33] caai: exec setuidgid user /local/bin/bitcoind 2>&1
[14:34] caai: export HOME=/home/user
[15:30] dorion: caai, mind using http://welshcomputing.com/paste/ in case it's something subtle that doesn't come through in the paste into chat buffer ?
[17:10] jfw: in this case not so subtle.
[17:12] jfw: caai: it was bitcoind that needed that HOME variable set
[17:12] jfw: I can point out that a script is a sequence of commands, i.e. in a particular order
[17:13] jfw: so what good is it going to do to set the variable *after* running bitcoind, hm? :)
[17:16] jfw: caai: it'd be like putting the key signature at the end of the movement!
Day changed to 2022-05-29
[15:00] caai: dorion; sure, i will use http://welshcomputing.com/paste/ in the future
[15:09] caai: jfw: nice analogy! I have placed the HOME variable before running bitcoind and it is up and running now! danke schon!
[15:55] caai: I will now attempt to finish the remaining homework from lesson 9
[17:53] jfw: caai: cool beans.
Day changed to 2022-05-31
[23:08] caai: jfw: I am signed into the meeting
[00:03] jfw: rodolfo: this urllib2 snippet for example even does a POST upload without any explicit socket operations
[00:08] jfw: oh hey, it's May.
[00:28] rodolfo: Understood about the technical questions. I'll try to figure out most things on my own, if I feel like on a dead end, I'll let you know.
[00:29] rodolfo: The list is great, I'll do my best best in getting to know those new to me.
[00:38] rodolfo: Regarding the socket config, I think I misread the initial docs since it was pointing out that urllib/urllib2 requires the *sockets* module. Both urllib versions are indeed high-level interfaces. Sorry about that
[00:49] rodolfo: jfw: Regarding an external server, I was thinking about using something like Heroku or Firebase (hypothetically) where I could set up a Python3 environment and make use of modules that facilitate the logic (ie receiving an input, doing inference and outputting 'the prediction').
[00:49] rodolfo:
[00:49] rodolfo: Then, exposing an endpoint from that instance so the input from the VM env will return the prediction. Nothing complex.
[00:49] rodolfo:
[00:49] rodolfo: Makes sense?
[00:50] rodolfo: Not sure about potential latency issues, but if so then we could keep everything of the VM by making use of the native functionalities, as I mentioned before
[00:54] jfw: rodolfo: it sounds indecisive possibly; like, you're trying to use two different systems so as to get the benefits of each, but it risks that you just end up paying the costs of each. we may need some clarification on the strategic perspective. dorion, any thoughts?
[00:56] jfw: perhaps it does make sense, as a first step from what you have now, just moving the more ...front-end parts of the back-end onto gales
[00:58] jfw: but if there isn't a longer term goal of getting the whole thing moved, it's less clear to me what would be the benefit to your project of using gales at all
[01:00] jfw: or more generally, of using a jwrd-hosted server if it's going to depend on 'the cloud' anyway
[01:09] rodolfo: Yeah, it is subject of further discussion. However, the only reason why I bring it up is just to save some time. I know no one has mentioned anything regarding deadlines but if time is not an issue, then I can dedicate whatever it takes to make it work on the current server.
[01:16] jfw: heh, well it might not make sense for *you* to allow unlimited time for the project, but I won't be the judge of that. as far as keeping the current VM online, the costs are pretty minimal because we'll be using its host server for other things too. as far as my development time, the idea is that it's for things that align with the direction we were going with Gales anyway, so again not a high
[01:16] jfw: marginal cost.
[01:19] jfw: that said, we do need to keep the lights on, we can't offer free things indefinitely and the point of this project as I understand is ultimately to get revenue; so there's going to be some implicit deadline at the very least.
[01:22] jfw: "implicit" as in the situation where there seems to be plenty of time until all of a sudden there isn't; and as yet I can't say where that point is but so far it's still in the "plenty of time".
[01:31] jfw: I could certainly hear an argument to minimizing the up-front technical wrangling, focusing on getting a sellable product first of all, then using the cash flow to support & justify backend optimization, as long as the current technical decisions aren't painting us into a future corner.
[01:47] rodolfo: Yeah, I think we are pretty aligned in terms of focusing on the goal which is the production of a tool that can be provided in exchange of an inflow in whatever currency expression. I do want to establish some time parameters, nothing that could become counter-productive but just for my overall mental health is good not to procrastinate. Let me figure out a few things and I will communicate you what these time parameters could look like.
[02:01] jfw: sounds good.
[18:05] rodolfo: Hi. While I figure out a set of deadlines, I'll share with you how I'm coming along with overall experience of translate the logic from Py3 to the VMs Py2 reality.
[18:08] rodolfo: For now, I'll do the data preprocessing on my local computer, this will result in a X by X array which then I will sftp to the VM. Then, using the native modules, I will write the necessary functions for the regression process that will later result in an input method for inference.
[18:09] rodolfo: X by X, N by N.. you get the point
[18:22] dorion: http://fixpoint.welshcomputing.com/2022/jwrd-logs-for-May-2022/#4069 -- I agree with you jfw. we went down this path of using Gales becasue rodolfo said his tool is portable to python 2. rodolfo, I think an important step in the planning is to document the parts of your tool that are python 3 only. e.g. are you using libaries that only exist in python 3 ?
[18:22] sourcerer: 2022-05-01 00:54:09 (#jwrd) jfw: rodolfo: it sounds indecisive possibly; like, you're trying to use two different systems so as to get the benefits of each, but it risks that you just end up paying the costs of each. we may need some clarification on the strategic perspective. dorion, any thoughts?
[18:26] rodolfo: @dorion: hi, yes but "tropicalization" is possible with creativity and effort hehehe
[18:43] dorion: not sure what you mean by tropicalization ? rewriting the python3 things in python2 ?
[18:46] rodolfo: For example, I'm solving a workaround for the One Hot Encoding method. This technique comes out of the box in sklearn, but jfw mentioned that the current Python 2 set up might not be suitable for numpy computation and sklearn relies on numpy. Having said that, there are options that not require any "magical solution" from Py3 modules, but would return the same result.
[18:48] rodolfo: It's not just translation, but some kind of reingeneering hehehe
[18:48] rodolfo: But that's OK, I was expecting this level of challenge
[18:50] rodolfo: With the Py2 native modules I have enough to do it, from what I can see
[18:58] dorion: ok, thanks for clarifying.
[19:00] rodolfo: Sorry for not bein explicitly clear. Maybe you guys were thinking about something like print() vs print kind of translation jaja
[19:35] jfw: well, let me expand re numpy, noting that I haven't really used it and am not really up to speed on the details of Machine Learning otherwise. my understanding is that it's basically a SIMD layer for Python, giving it adequate performance for large-scale numeric computations by adding homogeneous array data types and bulk operations on them. thus, as an interpreter extension, it's not going to be
[19:35] jfw: as simple as 'pip install numpy' or whatnot on Gales, unless it has a pure-python fallback though that would seem to defeat the purpose.
[19:39] jfw: it certainly existed back before the python3 push, so it's probably doable; though as yet we don't have experience with adding these third-party extensions into the python build.
[19:41] jfw: but, if we can do without for now, we may find that the performance isn't actually a problem at our current scale.
[19:42] jfw: my suggestion to rodolfo was just to try it and document what happens, For Science, even though I'm not expecting it to Just Work.
[19:43] rodolfo: Understood my friend
[19:46] jfw: rodolfo, have you done much work with python internals (eg the extension or embedding APIs, or just patching or compiling it)?
[19:48] jfw: or are you interested in such, since you're relying on python after all?
[19:53] rodolfo: Hi. I have experience with native Py modules but, since during work "time is money" I do have to rely on frameworks that ease the direct use of the native functionalities. But, overall, my work is about solving problems whatever it takes and I'm managing myself under the same premise within the current scenario.
[19:53] rodolfo: Fear not, I have everything under control
[19:53] rodolfo: =)
[19:54] rodolfo: Update: data is already on the VM
[19:56] jfw: heh, I wasn't afraid, merely exploring possibilities.
[19:59] jfw: I was asking about internals as in the C level, not merely the standard library which is what I guess you mean by 'native'.
[20:21] jfw: regarding frameworks, I'd say there's two ways they end up being used: one, as a form of leverage, when used by a skilled hand to increase productivity; two, as a crutch, when used out of ignorance or laziness to use simpler tools. In either case you pay the costs of adopting the framework, or worse, someone else down the line has to pay them, when it breaks or gets in the way or otherwise needs
[20:21] jfw: maintenance.
[20:23] jfw: that's why I'm not in favor of "teaching kids to code using python, it's so friendly"; let them start with machine code and work their way up to the power tools.
[20:43] rodolfo: Yeah, I'm considering some C/C++ possibilities but right now I might stay in the Py realm: after having all the data at the VM, I accomplished setting up some udf/modules to read the data accordingly.
[20:44] rodolfo: Next step - - > regression process
[20:48] rodolfo: To make my story a little bit sad, I never had the opportunity to actually learn any of this until I was 34 year-old. However, my previous experience was with extensive data analysis so it wasn't completely difficult for me to understand some concepts. My "tech lingo" comes from reading informal information sources, sorry if by communicating any message I don't do it properly. I'll try to do my best
Day changed to 2022-05-02
[00:24] jfw: rodolfo: hey no worries, you've got to start somewhere; and as a 32 year old I certainly hope 34 isn't the cutoff for being allowed to pick up new things!
[00:26] jfw: sounds like good progress on getting set up with the data. are you keeping notes on your steps, at minimum for yourself so you can repeat them later?
[00:55] rodolfo: Yeah man, I'm 37 yo and still learning a bit of everything
[00:56] rodolfo: Yes sir, I can replicate everything I'm currently doing
[18:20] rodolfo: UPDATE: I'm working on the calculations required for factor sensibilities.
[18:24] rodolfo: @jwrd could you explain how to use the "gports" method for installing add-ons?
[18:24] rodolfo: @jfw
[18:27] rodolfo: https://usercontent.irccloud-cdn.com/file/jKXTvRql/Screenshot_20220502_132659.jpg
[18:36] rodolfo: I think I can use pip directly in the terminal, but, like I mentioned before, PYTHONHOME (or similar Py path variable) is not defined yet. Is the Python Path = /gales/pkg/python-0.3-2.7.13/bin/python?
[18:37] rodolfo: Please confirm, thanks!
[18:59] jfw: rodolfo: I'd say you need to focus more on the "whats" and "whys" before jumping to the "how tos", especially since you're in an unfamiliar environment; otherwise how do you know that what you think you want to do is actually what's needed?
[19:07] jfw: to (perhaps annoyingly) answer exactly what was asked though, you install gports by obtaining the Gales repository and distfiles, entering the relevant subdirectory of "gports", running "gbuild" (preferably as the "build" user), then "gpkg-install -f" on the resulting package file; as explained in the PORTS file in the same repository. The path you show is indeed the concrete path to the python
[19:07] jfw: executable, but it's unclear what you mean by Python Path or why PYTHONHOME is necessary.
[19:07] rodolfo: I know this whole interaction with Gales is kind of an experiment (from my side), however I do believe some ease of use could become handy in order to replicate (sooner than later) what I did on a regular env. My point is that dealing with Gales shouldn't become a struggle for at least getting the basic functionalities (ie installing Py modules via pip). I don't think it adds any value to come up with my version of, for example, performing a t-distribution
[19:07] rodolfo: in order to find P-values for each factor. I'm not sure if I'm being clear enough, perhaps I can give RD a call in order to contextualize my comments.
[19:08] jfw: rodolfo: what would you say "pip" is for?
[19:09] rodolfo: The Py repository of packages/libraries
[19:09] jfw: well PyPI is not the same as "pip" though, is it?
[19:09] rodolfo: Let's say I need to install scipy, how do you recommend doing so
[19:10] rodolfo: It could be pip, or whatever other method. That's why I mentioned the gports you were referring to earlier
[19:12] rodolfo: It could be even from another repository
[19:13] jfw: well, I'd hazard a definition of pip as "a tool that automatically imports third-party code into your system, in an attempt to support the pretense that such code is external to your codebase and someone else's problem"
[19:13] jfw: in other words: in my view it's automating the wrong thing
[19:14] rodolfo: It's OK, let's say I just need the miracle (installing X library from a repository)
[19:14] jfw: so if I wanted to run scipy, my approach would be to look up scipy itself, find how to obtain it, check the docs, what versions are available, how to install etc.
[19:14] rodolfo: I understand
[19:16] rodolfo: In conclusion, there is no automatic way to get/install a library from a repository (in the exact same way I could achieve it with pip or similar methods). It should be done manually. Correct?
[19:17] jfw: I can't as yet say why pip isn't working for you; I didn't intentionally kill it or anything
[19:17] jfw: just that it's not very interesting to me, I suppose.
[19:17] rodolfo: Not saying that, I just want to have a clear view of the situation
[19:18] jfw: but I expect that doing things manually at least for the initial exploration would be enlightening
[19:19] jfw: meanwhile I've got the details on the apparent pip failure: http://welshcomputing.com/paste/suaatqhatu
[19:21] jfw: the exec_prefix/PYTHONHOME thing looks like a red herring - I don't expect there are any "platform dependent libraries" to load since it's a static build, perhaps it's a python or packaging bug - but the real problem is simply the "No module named pip". It's not installed!
[19:23] jfw: I expect "setuptools" at least will be required to install most python packages these days, even manually; and there is a gport for that, which I can install for you if that's what's needed.
[19:36] rodolfo: The "pip situation" can potentially be fixed with a wheel called "ensurepip" which is currently in the VM
[19:37] rodolfo: Let me explore
[19:43] jfw: miracles on wheels, that's what the market has come to expect these days.
[20:04] jfw: meanwhile in a total random coincidence, from what the market actually delivers: "We are working on restoring our services following a network outage. At the moment we are unable to process any orders. We apologize for the inconvenience." - major electronics components distributor
[20:31] rodolfo: Chaos is everywhere
Day changed to 2022-05-05
[16:50] dorion: hey rodolfo, for the mvp, we're thinking it's probably best to just stick with flask for the front end because a) that's what you already have b) apache isn't ready yet on gales and c) using php and apache is going to require some deep thinking on the architecture.. not that deep thinking is something to be avoided, just that it looks like a premature optimization and sticking with flask means the
[16:50] dorion: ability to cashflow it sooner.
[17:05] jfw: ah thanks for refreshing that point dorion. to expand a bit, it sounds like the current implementation does much of the work on-demand in python, so decoupling the 'data parts' from the 'web parts' isn't straightforward.
[17:06] jfw: and there's many input parameters so not well suited to pre-generating static html.
[17:07] jfw: the work to be done on the Gales porting front then would be getting the various numeric & other such libraries working
[17:08] jfw: and on the business logic side, implementing some type of subscriber authentication.
[18:30] jfw: http://fixpoint.welshcomputing.com/2022/the-passive-threads-that-would-like-to-be-given-some-blocks/
[21:39] jfw: aaand http://fixpoint.welshcomputing.com/2022/two-patches-and-a-new-fetch-bitcoind-release/
[21:42] jfw: (the article that was supposed to come first!)
[21:58] dorion: jfw, nice !
[21:58] dorion: hello donald, who might you be ?
Day changed to 2022-05-07
[01:26] jfw: I've added a browser-accessible gateway to the pastebin service, presented as the usual text box form input; so now it should be usable indeed from just about anywhere.
[01:28] jfw: surprisingly the php sockets functions weren't enabled by default, so I added --enable-sockets to the ./configure list in the php gport.
[01:30] jfw: (they're used for connecting to the backend which is the same plain TCP responder on port 4.)
Day changed to 2022-05-08
[12:58] caai: jfw: Could you verify that I am creating the run script for /etc/svc/bitcoind correctly? My first estimation is:
[12:58] caai: #!/bin/sh
[12:59] caai: exec bitcoind /home/user/.bitcoind
[20:15] donald: Justin - also known as donald or jwm
[20:17] donald: hexchat is a bit funky, didnt remember my original login - ubuntu is special!
[20:18] jfw: indeed, though I suspect the 'snap' thing that wanted to be cooler than apt contributed to the tangle.
[20:22] jfw: caai: good start, but bitcoind doesn't take a directory argument like that, does it? I'm guessing the reason you wanted to put it there is valid though
[20:23] jfw: if so, what do you think the underlying problem is?
[20:34] caai: well, in order to start bitcoind, no argument is added, so I understand my logical misstep.
[20:35] caai: On the other hand, bitcoind must be executed within the .bitcoind directory, therefore:
[20:35] caai: #!/bin/sh
[20:36] caai: exec /home/user/.bitcoind/bitcoind
[20:37] caai: ?
[20:37] caai: however, there is no 'file' named bitcoind in the .bitcoind directory
Day changed to 2022-05-09
[02:18] caai: which bitcoind indicates: /local/bin/bitcoind
[04:33] jfw: caai: that would be one sense of "executed within" I suppose - where the executable itself is located in that directory; though perhaps you have in mind the other sense, of being the working directory in which the process is run. (recall program is to process something like what dna is to individual organism)
[04:39] jfw: but, I don't believe it has to be executed in any particular working directory. It does however have to be run as the correct user, and since the service supervisors run initially as root, perhaps this is at the ...root of your trouble?
[04:40] jfw: if you want to set the working directory, you'd add a "cd" command; it's simply scripting the same commands you'd run interactively at the terminal.
[04:42] jfw: to set the user, perhaps you could mess around with "su" but daemontools includes a more handy thing for the purpose, setuidgid
[04:44] jfw: which gets us to something like: "exec setuidgid user /local/bin/bitcoind"
[04:44] jfw: next, I'd redirect the command's standard error into its standard output so they both get captured by the log sub-service, if any: "exec setuidgid user /local/bin/bitcoind 2>&1"
[04:47] jfw: finally, I see in my own run script I'm also setting up the HOME environment variable, presumably because setuidgid is too narrowly specified to take care of such conveniences, and bitcoind needs it to find the correct home directory.
[04:49] jfw: I'm mildly curious to see what happens without it though, so I'll leave that part as an exercise :)
[19:43] dorion: http://fixpoint.welshcomputing.com/2022/jwrd-logs-for-May-2022/#4170 -- hey, welcome back.
[19:43] sourcerer: 2022-05-08 20:15:08 (#jwrd) donald: Justin - also known as donald or jwm
Day changed to 2022-05-10
[11:49] caai: jfw: I created the 2 lines in the 'run' file as inidicated: "exec setuidgid user /local/bin/bitcoind" and "exec setuidgid user /local/bin/bitcoind 2>&1"
[11:51] caai: next, 'chmod +x /etc/svc/bitcoind/run'
[11:52] caai: then, svc -u /services/bitcoind
[11:53] caai: (perhaps that was unnecessary and it would be up by default)
[11:59] caai: finally, upon running 'svstat /services/bitcoind', I see that the pid changes each time I run it and that the service is up but only for 1 second on average, thus, indicating that it is going up and down
[17:03] jfw: caai: I didn't mean two separate lines, rather was building up the one line by parts, so just keep the second. then, you're correct, it's starting then immediately exiting; so to find out why, you have two log files to check: the standard output/error captured in /var/log/bitcoind/current, or the usual ~/.bitcoin/debug.log.
Day changed to 2022-05-13
[14:00] caai: jfw: I see. I have reduced the script to the second line
[14:07] caai: next, I stopped the node via 'kill -9', then i 'tail -f ~/.bitcoind/debug.log' | 'svc -u /services/bitcoind'
[14:11] caai: 'tail -f ~/.bitcoin/debug.log'*
[14:12] caai: as I observe result of (tail -f ~/.bitcoin/debug.log), I see no output
[14:13] caai: when I 'ls /var/log/' the output is klog and sshd. I see no directory for bitcoind
[20:02] caai: cleary, i am missing something .....
Day changed to 2022-05-14
[05:08] caai: clearly*
[14:34] rodolfo: Good day friends!
[14:35] rodolfo: @dorion today I'm taking my FRM level 1 exam, I'll be out around 2:30 pm
[15:17] rodolfo: Man, I really live at a fucking jungle.
[15:18] rodolfo: The testing center forgot to open their facilities today.
Day changed to 2022-05-16
[18:48] jfw: caai: I'll guess that there's no log sub-service defined; you can check this via 'svstat /service/bitcoind/log'. if that's the case, you can create one using the generic/default log service from gales: mkdir -p /etc/svc/bitcoind/log ; ln -s /etc/svc.defs/multilog /etc/svc/bitcoind/log/run
[18:48] jfw: once that comes up, /var/log/bitcoind should be created and start capturing whatever's going wrong.
[18:49] jfw: rodolfo: ouch, sounds very LatAm/tropical indeed.
[18:50] jfw: that's a Financial Risk Manager exam I take it
Day changed to 2022-05-17
[13:24] caai: jfw: upon running 'svstat /service/bitcoind/log', I saw that there was no log file. I created one running 'mkdir -p /etc/svc/bitcoind/log ; ln -s /etc/svc.defs/multilog /etc/svc/bitcoind/log/run'
[14:31] caai: I then ran 'svc -u /service/bitcoind'. After that, I ran 'ls /var/log/'. The 2 directories displayed are klog and sshd, but no bitcoind file appears
[14:33] caai: I will continue to investigate this matter tomorrow
Day changed to 2022-05-18
[04:47] jfw: caai: from some testing and reading between the lines of the svscan docs it seems this case of a log service being added after initial supervisor startup isn't really provided for. thus to remedy it, short of rebooting the machine, we'll have to fully disable the service then re-trigger it.
[04:48] jfw: 1) rm /service/bitcoind [to prevent svscan from restarting the un-logged supervisor]
[04:48] jfw: 2) svc -x /etc/svc/bitcoind [to signal the un-logged supervisor to bring down the daemon then exit itself]
[04:49] jfw: 3) ln -s /etc/svc/bitcoind /service/ [to re-enable so svscan will start the now-logged supervisor]
[04:53] jfw: within 5 seconds, 'svstat /service/bitcoind/log' should indicate up and the /var/log/bitcoind directory should be created.
[14:49] caai: jfw: I followed all of the above steps and the /var/log/bitcoind directory was created. Thank you! I will now search in the log to discover what I can
[14:55] dorion: caai, nice !
[18:18] jfw wonders if "work out how to create your own daemontools service for bitcoind" was too much for a homework assignment. perhaps just too much for a *first* unguided one in that direction
Day changed to 2022-05-20
[11:22] caai: jfw: when i 'tail /var/log/bitboind/current' the most relevant information i see in the error message is:
[11:23] caai: 'tail /var/log/bitcoind/current'*
[11:30] caai: [what() boost::filesystem::create_directory: Permission denied: "/.bitcoin"]
[11:58] caai: I imagine this error is due to not having file system access due to not having user privileges. Since all of the service files have a uid and gid of 'root', that seems odd because root has unlimited privileges.
[12:05] caai: however, in the script '/etc/svc/bitcoind/run' setuidgid is 'user'
[12:17] caai: I setuidgid to 'root' and it seems to be executing properly now
[12:25] caai: it is not running bitcoind yet, but the logs are showing some configurations that i need to change in /.bitcoin/bitcoin.conf
[12:28] dorion: caai, what's in /.bitcoin ? compare that with /home/user/.bitcoin
[12:32] dorion: http://fixpoint.welshcomputing.com/2022/jwrd-logs-for-May-2022/#4229 -- I'm not surprised that it's executing now, but suspect it's not proper.
[12:32] sourcerer: 2022-05-20 12:17:15 (#jwrd) caai: I setuidgid to 'root' and it seems to be executing properly now
[12:38] dorion: might have something to do with : http://fixpoint.welshcomputing.com/2022/jwrd-logs-for-May-2022/#4188 http://fixpoint.welshcomputing.com/2022/jwrd-logs-for-May-2022/#4189
[12:38] sourcerer: 2022-05-09 04:47:32 (#jwrd) jfw: finally, I see in my own run script I'm also setting up the HOME environment variable, presumably because setuidgid is too narrowly specified to take care of such conveniences, and bitcoind needs it to find the correct home directory.
[12:38] sourcerer: 2022-05-09 04:49:40 (#jwrd) jfw: I'm mildly curious to see what happens without it though, so I'll leave that part as an exercise :)
[16:55] jfw: caai: dorion has it; we don't want to run it with unlimited privileges but continue running as 'user' as before, that's why the setuidgid. so the problem is that it's trying to use /.bitcoin for data dir rather than the correct /home/user/.bitcoin that we already set up, and that's because of the missing HOME environment variable.
Day changed to 2022-05-21
[14:28] caai: I see. I have edited the script to 'setuidgid user'. I am now researching how to set environment variables.
[22:42] jfw: Sound the trumpets, beat the drums...
[22:42] jfw: http://fixpoint.welshcomputing.com/2022/sending-buffer-overflows-fixed-in-bitcoind-and-other-cleanups/
Day changed to 2022-05-22
[00:11] dorion: jfw, well done, quite the enjoyable read there. I liked the inclusion of the code in the article for this one. You added many comments in the code to so there were a few levels of commentary going on.
[04:53] jfw: thanks dorion, indeed there were multiple levels there.
[05:26] jfw: ah, http://billymg.com/2022/01/building-trb-on-a-2022-vintage-musl-gentoo/ was the ref I was looking for at http://fixpoint.welshcomputing.com/2022/jwrd-logs-for-Apr-2022/#3850
[05:26] sourcerer: 2022-04-18 17:05:32 (#jwrd) jfw: http://welshcomputing.com/paste/hhz52895bm << error encountered in building bitcoind on a stock ubuntu with gcc "10.3.0", coming from improper redefinition of __atomic_compare_exchange. I swear I'd seen this before but haven't turned up any discussion.
[05:27] jfw: still need to decide on and integrate a proper solution to these.
[05:35] jfw: that, and/or do a recipe for building in a Gales chroot.
[05:39] jfw: but for the 'let me just try this out on ubuntu without bootstrapping a whole new system' use case that might be overkill.
[05:40] jfw: a low priority perhaps, but the code is technically being stupid and ideally would get fixed.
Day changed to 2022-05-23
[18:57] caai: jfw/dorion: shall we have the next management meeting on: May 31 - 23:00 UTC? | I think that I have to make more progress with my assignments before the scheduling the next Operator Meeting
[18:57] jfw: caai, that should work for me
Day changed to 2022-05-25
[12:41] caai: jfw: I believe I am aproximating a line with the correct syntax to set home environment variable. I have tried numerous variations of "export PATH=etc.." to no avail
[12:44] caai: At the moment, I am reading the man page for ksh to learn what I can
[13:46] dorion: caai, you're trying to set HOME rather than PATH, right ?
[17:45] jfw: ^
[17:46] jfw: as strong a move as reading the ksh man page may be, looks like this more basic confusion.
Day changed to 2022-05-27
[13:21] caai: alright. my next guess was to try "export HOME=/home/user", which was unsuccessful. After that, I tried "export HOME=/home/user:$HOME", which was also unsuccessful
[17:59] jfw: caai: the first one there should be correct so perhaps something else is wrong; maybe paste the full "run" script you came up with?
Day changed to 2022-05-28
[14:30] caai: alright. here is the full 'run' script that I came up with:
[14:31] caai: #!/bin/sh
[14:33] caai: exec setuidgid user /local/bin/bitcoind 2>&1
[14:34] caai: export HOME=/home/user
[15:30] dorion: caai, mind using http://welshcomputing.com/paste/ in case it's something subtle that doesn't come through in the paste into chat buffer ?
[17:10] jfw: in this case not so subtle.
[17:12] jfw: caai: it was bitcoind that needed that HOME variable set
[17:12] jfw: I can point out that a script is a sequence of commands, i.e. in a particular order
[17:13] jfw: so what good is it going to do to set the variable *after* running bitcoind, hm? :)
[17:16] jfw: caai: it'd be like putting the key signature at the end of the movement!
Day changed to 2022-05-29
[15:00] caai: dorion; sure, i will use http://welshcomputing.com/paste/ in the future
[15:09] caai: jfw: nice analogy! I have placed the HOME variable before running bitcoind and it is up and running now! danke schon!
[15:55] caai: I will now attempt to finish the remaining homework from lesson 9
[17:53] jfw: caai: cool beans.
Day changed to 2022-05-31
[23:08] caai: jfw: I am signed into the meeting