General questions

[okynos]

Do you have some questions? We will solve any related doubt.

[thisnugroho]

Hello, first of all, thank you for putting effort into this project. It's really impressive.

I apologize if I'm not in the right place to ask this, but I have some questions. Is it possible to find out who created or edited the file?

[okynos]

Hello @thisnugroho,

Hello, first of all, thank you for putting effort into this project. It's really impressive.

Thanks for the words, really appreciated.

I apologize if I'm not in the right place to ask this, but I have some questions. Is it possible to find out who created or edited the file?

No worries. I created FIM project and all the files that you can see on GitHub. Do you have any problem or maybe you want to edit something? What file do you mind? I can help you with it.

[thisnugroho]

You're welcome 😉

{
  "checksum": "a69f73cca23a9ac5c8b567dc185a756e97c982164fe25859e0d1dcc1475c80a615b2123af1f5f94c11e3e9402 c3ac558f500199d95b6d3e301758586281dcd26",
  "detailed_operation": "CREATE_FILE",
  "file": "/var/www/shell.php",
  "fpid": 16662,
  "hostname": "localhost",
  "id": "c9d7ee84-1d96-41d5-9e39-2791592ee153",
  "labels": [],
  "node": "FIM",
  "operation": "CREATE",
  "system": "linux",
  "timestamp": "1698591757187",
  "version": "0.4.9"
}

I am looking for the following details from the logs:

1. Which file was created, edited or deleted? ✅
2. When did the action occur? ✅
3. Who created/edited/deleted the file? ❓
4. Size of the file ? ❓

Do you know how I can get an answer from number 3 and 4 ?

[okynos]

3. Who created/edited/deleted the file? ❓

To know who performed the action you will need to enable the Audit feature and set up the required directory to monitor. The system requires the Audit to provide user information. The default inotify feature does not provide such information. Take a look at https://documentation.achiefs.com/docs/configuration-file.html#audit for more information.

4. Size of the file ? ❓

Currently, FIM does not provide the file size of an event. But I have to say that I have reviewed the code, and it is not hard to include. I will include it as soon as possible.

[thisnugroho]

Hello, I attempted to install it on my other server, but it gave me an error message.

dpkg: error processing archive fim.deb (--install):
 package architecture (amd64) does not match system (arm64)
Errors were encountered while processing:
 fim.deb

I need usingHetzner server, which has an ARM64 architecture. Can you help me with this?

[okynos]

Hello @thisnugroho,

I'm struggling with an AARCH64 architecture compilation, for this reason I did not upload the RPM and Debian ARM packages.
I have compiled this one meanwhile:
DEB ARM64
Give it a try and let me know anything you need. Thanks!

[thisnugroho]

Hello,

Cool it's working now, Thank you so much 👍 😄

[okynos]

Hello @thisnugroho we have published officially the v0.4.10 if you wish to test it out. Thanks.

[thisnugroho]

Thank you so much ✨

[okynos]

Hello @thisnugroho, I hope all is going fine. I merged the #134 issue into the main branch (0.5.0). I want to ask you. Have you any requirements or feature requests that you may require to include in 0.5? Feel free to suggest anything you need. You can also check the issues list at https://github.com/Achiefs/fim/issues.

Thanks in advance, and have a nice day!

[thisnugroho]

Thank you so much,

i was wondering Is it a good idea to set notifications for when files that meet specific rules are created or exist? Or do you have any recommendations on how to accomplish that?

[okynos]

Do you mean something inside the document? Or something that surrounds the file? Do you want to produce an event when some regex is inside a document or if one file obtains execution permissions, plus it is executed?
In summary, could you give me some examples? Thanks!

[thisnugroho]

for example If there is a new file with the extension *.php in a specific folder, I would like to receive a notification.

[okynos]

Ok I got that. I will create an issue to include it in 0.5. Feel free to include any corrections when the issue is ready.

Again, thanks for your time and words!

[thisnugroho]

Sure, thank you so much

You are so amazing ✨ !

[zbalkan]

Do you have plans to provide a default backend server? Or do you want to keep this as a host-only solution?

[zbalkan]

I use Wazuh for all my needs. But there are some friends an acquaintances in need of a FIM, while keeping their SIEM in place. Therefore, this tool may help by either sending logs to the SIEM agent locally, or building a backend and alerting on top of it. You mentioned elasticsearch and OpenSearch. These days I am testing Quickwit, a rust based immutable Document database. It looks less stable but more lightweight than other options. Instead of sending all logs to SIEM, it would make sende to add a middleman (elasticsearch, OpenSearch or Quickwit) and sending only alerts triggered by them to SIEM would make things simpler. I have many FIM rules in Wazuh, but they are actually not security rules. They are just filtering important file changes. Delegating this filtering to a database would make more sense. And it would also minimize the SIEM costs since many licenses are calculated by EPS or storage.

[okynos]

So If I understand it. You suggest something in between OpenSearch and Wazuh to filter related security events? and then only take action where specific events are received at the OpenSearch level? OpenSearch can't forward events right?
Maybe we will need a backend runtime that receives the events and produce a security event. Thinking on it FIM will only produce events you may want to know (Monitoring critical paths) then you can take a decision when you receive the event at OpenSearch stage.
Maybe I sound confused. It's just because I'm brainstorming on it.

[zbalkan]

I mean something dedicated like OpenSearch. People spend a lot for a SIEM, for instance Splunk. So they don't want to change that. Something in between Splunk and FIM, that would store the change logs, but send only alerts matching some rules. For instance they may just need /etc for configurations, /opt for their own applications on Linux, etc. I have rules for system cryptography settings in RHEL, for instance. Then, only these alerts may be sent to SIEM, while other logs would be kept in the OpenSearch, Elasticsearch instance. Do you plan to leave this to the users to build own their stack? Or do you want to have a preconfigured setup?

[okynos]

Now I understand better the idea.
I know the users like to manage the events received. Your request will allow the users to process only the critical events at the indexer level.
However, I think the development of a SIEM will be out of the scope of this project and will need a common structure.
I have been thinking about that. How does it sound to you to have a package called FIM (File monitoring), another called EventResponse, another called LogReader (process logging) even another called Receiver (to process any module received events)?. With the mentioned schema we could manage users' events and then send them to the indexer.
I could develop those modules each one with separated schema and repository.

[zbalkan]

You don't have to send to Indexer directly. If you already have an agent, let's say Wazuh, we can read read FIM events whatever location configuration supports. That'd solve the log forwarding problem but not the filtering problem. That must be solved locally then. It became a configuration management issue for the user. User can use Ansible, Puppet, Chef or Group Policy to deploy rules.yaml to the endpoints. The SIEM of the user can handle the rest. If that's the design decision, it's better to write documents for users to help integration for forwarding FIM to SIEM agent, and rules/queries for FIM logs in Wazuh, Splunk, Elasticsearch+Elastalert, etc.

The other way requires a lightweight SIEM. It would simplify the life for users but it will require development efforts.

[nicolascharbonnier]

HowTo change date format ? (Linux)
i see : /etc/fim/index_template.json
"timestamp": { "type": "date" },

is it possible to get it readable human format ?

[okynos]

Hello @nicolascharbonnier

When you ingest an event into OpenSearch you will get a human readable format, I attach you an example:

If you will process the events.json file directly you can use any of Linux date conversion commands because it is a standard Unix timestamp.
Tell me if it helps you.

[nicolascharbonnier]

Hey, thx for reply so fast.
i don't use OpenSearch (for the moment) but i think, i will.

[zodiacnet]

Hi everybody,
first of all, FIM its a nice application!, I've used others on the past but had high impact on the disk or cpu, so far the short time I've used it seems to run very smooth and without much impact!, the reasom I'm here today is because I'm trying to enable audit feature, I've already installed auditd and confirmed that auditctl it's installed, but when I restart FIM it says
Auditctl command error: No such file or directory (os error 2)
I've reviewed what command tries to run when the error happens and seems like it's running "which auditctl" to find the auditctl path, I've even added the PATH env var to the service but I had no luck, do you have an idea about how to resolve this issue?
thank you in advance!

[zodiacnet]

solved!, just reviewed the code and was looking it on /usr/sbin/auditctl and on my system was on /sbin/auditctl so just created a symlink and it worked!

[okynos]

Hello @zodiacnet

I'm glad that you solved it.
Thanks for your feedback, which system do you use? I want to review the error.

[zodiacnet]

oh @okynos I forgot to add that detail on the ticket!
I was using Ubuntu 18.04.6 LTS (pretty old hehe)

[zbalkan]

Hi all. It's me again. Is there any updates on the project? I see some commits but the pace is slower. I wonder the future plans.

[okynos]

Hello @zbalkan,

Thank you for taking the time and interest in the project. Yes, we have plans and we are working on include some features in the near future to FIM. With near future I mean the next quarter, surely it will bump to 0.6.0 or even higher.
Those features are specifically two: Audit data on Windows (I know it is taking time to be ready but Windows requires a ton of specific code and dedication) and hash integrity scan and database, another big feature that is near to publication.
These two features are quite interesting at the same time tough to code, adapt and test of course.
On top of that we added some projects out of the technology stack as office renovation and some equipment upgrade too.

So, please keep your interest on FIM you will hear from our side pretty soon.
Thanks to all interested on FIM too.

PD: The development branches if you want to take a look:

• https://github.com/Achiefs/fim/tree/177-db-hash
• https://github.com/Achiefs/fim/tree/82-add-audit-windows

[zbalkan]

Hi @okynos,

Thanks for the response. I recently decided to work on my 2-year-old code again and remembered our discussion in the repository. I see you are using the most practical approach, also the one Wazuh used. I was trying to test 3 alternatives:

1. Event log channel: requires Local or group Policy changes for auditing, along with SACL changes.
2. FileSystemWatcher class: Dotnet-specific but possibly replicable. Simple, just works. Filtering is primitive. Lacking the user data.
3. ETW channel: Too noisy. No real-time listening. Requires exporting logs as small batches to a buffer before filtering as data consumption time should be sorter than data production time. It is easy to lose data.

Did you have time to benchmark all options?

[okynos]

In some point of the designing I remember those options. we should discard point 2 and 3 just for requirements reasons.
On point 2 as you mentioned it lacks of some data that looks really important from security perspective.
On point 3 comes from the real-time part, in our use case the reporting should be as nearly as real-time possible and it requires to manage queues. In any case, I encourage you to try and let us know pros and cons of each one. We will try to benchmark these options in some point of time but currently we are out of time for topics not related to produce a stable and a good code. Anyway, thanks for your suggestions there are always welcomed.

[zbalkan]

Hi. I did try different versions. ETW makes things very complex. I had to keep a thread-safe queue for logs in the main thread, an ETW listener in a separate thread and queue consumer in another thread. Even then, I saw there are losses. FilesystemWatcher is more reliable but it is dotnet specific but it uses ReadDirectoryChangesW, just like notify crate. I will have to go back to option one as well.

The benchmark is hard as Windows does so many things at the background. It makes a fair benchmark nearly impossible. Especially for ETW, since we do filtering after retrieving as much as we can in a buffer. It is so noisy.

At one point, I wanted to build my version of FileSystemWatcher with a larger internal buffer, similar to others but it was not a match for the original one again.

[Louisonix]

Is it triggering an event/callback like onBeforeFilesIntegryChange, or is it merely notifiying about changes that already happened? If so, is it possible to implement? Also, the event handlers should be able to cancel the file write. That's what's required for a GUI with options to either accept or discard the changes.

[okynos]

Hello @Louisonix,

FIM in the current version, 0.5.2, notifies when an action is performed to the file, like write, attribute change, move or remove on Windows. We are working on include additional information.
On Linux using audit you can get more information like file read or executions performed on the file and probably detects previous steps like onBeforeFilesIntegryChange. I'm not sure if it solves your doubts in any case it would be great to get more information on your specific use case.

Thanks for your feedback.

[zbalkan]

@Louisonix, may I ask what problem you are trying to solve?

[Louisonix]

For Windows 95/98/XP there once came a file integry monitor with the virus scanner "the cleaner". Everytime a process wanted to change files, it opened a popup asking if one would like to accept the changes (it also handled large chunks of file changes). I'm thinking about implementing something similar for linux, and stumbled upon fim when I searched for libs/programs required to implement something like that.

[zbalkan]

In Windows, that'd require a mini filter intercepting filesystem calls. In Linux, it requires intercepting all fopen, open and similar syscalls, either using eBPF or via a driver. The syscall/API tool to intercept depends on if you want to know at access time or write time.

FIM, as the name addresses, is a monitoring tool. I do not mean Achiefs FIM but all File Integrity Monitoring tools as a category. It is not proactive but reactive.

At this point, what you asked is building a Just-in-time access approval workflow on top of FIM. For that purpose, a FIM utility is not what you seek for. Also, I am not aware of similar tools regarding that capability.

[Louisonix]

Thank you for your more in-depth answer. Not sure if really want to hook the kernel or process functions, but I like the Idea. Would have been to easy if Linux inotify already provides such actions. On the other hand, I'm missing such an tool in the linux world and I'm wondering why there is no such tool for linux desktops. Maybe someone wants to step in :-D.

Greets