planning poker

January 9, 2015
By administrator

  • The skills and talents of individual programmers are the main determinant of software quality. No amount of management, methodology, or high-level architecture astronautism can compensate for a poor quality team.

  • The motivation and empowerment of programmers has a direct and strong relationship to the quality of the software.

  • Hard deadlines, especially micro-deadlines will result in poor quality software that will take longer to deliver.

  • The consequences of poor design decisions multiply rapidly.

  • It will usually take multiple attempts to arrive at a viable design.

  • You should make it easy to throw away code and start again.

  • Latency kills. Short feedback loops to measurable outcomes create good software.

  • Estimates are guess-timates; they are mostly useless. There is a geometric relationship between the length of an estimate and its inaccuracy.

  • Software does not scale. Software teams do not scale. Architecture should be as much about enabling small teams to work on small components as the technical requirements of the software.

-> http://mikehadlow.blogspot.sg/2014/03/coconut-headphones-why-agile-has-failed.html?new

Read More »

Logging, logging, logging

January 9, 2015
By administrator

karterk 10 hours ago | link

The most difficult thing is going to be getting to 10K active users :)
These days RAM is cheap and SSD storage is also widely available. For a very long time, one of my side projects with 50K users was hosted in a EC2 small instance. With that out of the way, here are a few things you will need to take care of:

  • Security (especially passwords) - Rails should take care of most of this for you, but you should ensure that you patch vulnerabilities when they are discovered. Also, stuff like having only key-based login to your servers etc.
  • Backups - Take regular backups of all user data. It's also VERY important that you actually try restoring the data as well, as it's quite possible that backups are not occurring properly.
  • One click deployment - Use Capistrano or Fabric to automate your deployments.
  • A good feedback/support system - this could even be email to begin with (depending on the volume you expect), but it should be accessible.
  • Unit tests - as your app grows in complexity, you will never be able to test all the features manually. I'm not a big fan of test driven development, but really, start writing unit tests as soon as you have validated your product idea.
  • Alerts, monitoring and handling downtime - Downtimes are inevitable. Your host or DNS could go down, you might run out of disk space, etc. Use something like Pingdom to alert you of such failures.
  • Logging, logging, logging - I can't stress on this enough. When things break, logging is crucial in piecing together what happened. Use log rotation to archive old logs so they don't hog the disk space. reply

icehawk219 9 hours ago | link

Backups - Take regular backups of all user data. It's also VERY important that you actually try restoring the data as well, as it's quite possible that backups are not occurring properly. The part about testing your backups is huge. I can't count how many projects I've been on that had problems where we needed to restore and we looked only to find any number of problems. Oh, backups actually stopped last month when we ran out of space, oops the backups only backed up these 3 db's and not the one you want, things like that. I'd also stress the importance of off-site backups. If you're using AWS for everything and your account is compromised can they delete your backups (assuming they have full, 100% unlimited, admin access to AWS)?
Which is also why if you're using stuff like AWS, Heroku, or any other third party provider (hosted Mongo, hosted ElasticSearch, Stripe, NewRelic, etc.) it's very important to ensure those passwords are secured and only the people absolutely necessary have access. Also, when offered, two-factor authentication should always be used.
reply

Rapzid 8 hours ago | link

And don't use keys on your console admin accounts.
reply

JoshTriplett 8 hours ago | link

  • Logging, logging, logging - I can't stress on this enough. When things break, logging is crucial in piecing together what happened. Use log rotation to archive old logs so they don't hog the disk space. Depending on the service you're building, you can log too much. Consider the privacy and security implications of the existence of those logs; anything you log can be subpoenaed, but logs that don't exist cannot be.
    Consider anonymizing your logs from day 1, and only turning on non-anonymous logging upon a report from a user. Alternatively, give users a "report a problem" button, and save their last N minutes of otherwise-ephemeral logs only when they hit that button.
    You absolutely want to log enough to help you debug the service, but do you really need to archive old logs, or should you delete them entirely?
    reply

landr0id 8 hours ago | link

Logging, logging, logging - I can't stress on this enough. When things break, logging is crucial in piecing together what happened. Use log rotation to archive old logs so they don't hog the disk space. How do most people manage activity logs? Currently what we have set up is the user id (if the user is logged in), IP address, URL they hit, user agent, and timestamp are all inserted into an activity logs table. For one particular site with an API that's being polled the size of the DB grew pretty large.
reply

oisinkim 3 hours ago | link

Logging, logging, logging - I can't stress on this enough. There is no easier way to offload, view, filter, alert and search than logentries:
http://www.logentries.com
reply

chrissnell 7 hours ago | link

Off-machine logging. There are commercial services (we're using Papertrail but there are tons of them), roll-your-own solutions (Elasticsearch-Logstash-Kibana), and simple solutions (syslog).
For an easy and simple solution, spin up a second instance and send logs to it via rsyslog over a private network interface. Most mature frameworks provide a method to send logs over syslog. It's UDP and very lightweight. Another plus: if you are compromised, you have another server with your logs and that server isn't running your vulnerable app.
reply

clogston 6 hours ago | link

We're running the Elasticsearch, Logstash, Kibana (ELK) stack with the recommended approach i.e.:
logstash client \ logstash client --> redis -> logstash "server" process -> elasticsearch <- kibana / logstash client
We have a high logging load (we log every request) due largely to IRS requirements. I've been really happy with it over the past 6 months but something that cannot be overstated is that you'll really need to become familiar with each one of the technologies used as each requires it's own setup and configuration. Not being familiar with any of them, it took me a solid 3 days to get to where the whole thing was usable and performant. Troubleshooting it is a breeze, and the whole system scales really easily, but a lot of that time was invested up front.
reply

daigoba66 8 hours ago | link

Logging every hit will always require a lot of space. But there are some tricks you can use to "compress" it: hash long strings like the URL and user agent and store the hash as binary instead of a string. A 100+ byte string can compress to just 16 or 32 bytes depending the hash your pick. Store the hash lookup in a separate table.
reply

mandeepj 4 hours ago | link

what is the benefit of your approach? The lookup table will still have data growth issues
reply

tonglil 6 hours ago | link

We keep the last 30 days and also send it out to Rollbar for notifications and analysis. It's working great!
reply

porker 10 hours ago | link

  • Logging, logging, logging - I can't stress on this enough. When things break, logging is crucial in piecing together what happened. Use log rotation to archive old logs so they don't hog the disk space. +1 You can't log too much. The user who claims an important email never arrived - does your system say it was sent? This bug 3 users have reported yet no one can reproduce - what were they doing at the time and what else was going on? No, I'm not at that stage yet (of effectively being able to rewind application state in the log files to see what was going on), but for debugging issues in production it's exceedingly useful.
    reply

edmack 9 hours ago | link

Getting loads of core services out into third parties is really wonderful for logging. E.g. if email sending happens in Mandrill, then you never need to write decent logging calls for that and you have a reliable source of truth!
reply

tlack 8 hours ago | link

Except you won't know if your server ever sent it to Mandrill. :) Always be extremely verbose with logging!
reply

porker 3 hours ago | link

This brings up a tangential problem I've yet to solve: how do you warn that something didn't happen when it should?
E.g. you have a script that does backups. You log the script's output, but one day something fails and the script is no longer executed.
Some form of dead man's handle is needed; the only way I can think of is to set up a monitoring service to check your log store for these entries every X hours.
Any alternatives?
reply

reymus 8 hours ago | link

I have always heard the opposite, that too much logging is as bad as no logging. I see the point of having the logs to be able to find out what happened, but what happens when there' s so much logging that the information needed is just buried into huge amount of noise?
reply

msielski 7 hours ago | link

This is true, without the right tools. I am moving to logstash with kibana to do this, and it's looking very promising. See http://www.elasticsearch.org/videos/kibana-logstash/
reply

revo13 5 hours ago | link

Concur with this. Log everything, and use Logstash/Kibana to sift through it for what you are looking for.
reply

exelius 6 hours ago | link

This was true before Splunk. If you logged too much, your logs could start to outstrip the assumptions behind your log rotations and cause trouble. Now the common wisdom is to just log everything so you can Splunk it later if you have a problem. Verbose logging + Splunk have made production incident identification so much easier than it used to be.
Splunk DOES charge by the GB, but it's not very expensive in the long run.
reply

gizmo686 7 hours ago | link

My favorite systems to work with are the ones with overly verbose logs, where the overly verbose parts were clearly tagged and could be filtered out. Generally, we would never look at the verbose lines, and even when we did, we would normally have some idea what we were looking for, and be able to filter somewhat for it. reply

icehawk219 8 hours ago | link

I'd actually argue it is possible to log too much if you aren't using good tools to make your logs easily searchable. Which is why you should use such tools if at all possible. Otherwise the logs can become so big that finding the entries for that bug or that email becomes pretty much impossible. This is also why it's important to take a few minutes and think about what you're logging and how. Things like request and user IDs can be invaluable. My test is usually "if I have nothing but this error message/log entry, do I have enough to begin investigating?". This is hard to get right until a bug or problem occurs and you actually need to use the logs but investing a bit of time into it can be a life saver.
reply

Read More »

  • Untitled-Scanned-99
  • Untitled-Scanned-95
  • Untitled-Scanned-92
  • Untitled-Scanned-68
  • Untitled-Scanned-93
  • Untitled-Scanned-91
  • Untitled-Scanned-59
  • Untitled-Scanned-86
  • Untitled-Scanned-65
  • Untitled-Scanned-67
  • Untitled-Scanned-63
  • Untitled-Scanned-66
  • Untitled-Scanned-64
  • Untitled-Scanned-58
  • Untitled-Scanned-57